[jira] [Created] (YARN-10535) Make changes in queue placement policy to use auto-queue-placement API in CapacityScheduler

[Wangda Tan (Jira)]

Wangda Tan created YARN-10535:
-

 Summary: Make changes in queue placement policy to use 
auto-queue-placement API in CapacityScheduler
 Key: YARN-10535
 URL: https://issues.apache.org/jira/browse/YARN-10535
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: capacity scheduler
Reporter: Wangda Tan


Once YARN-10506 is done, we need to call the API from the queue placement 
policy to create queues. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used

[Wangda Tan (Jira)]

Wangda Tan created YARN-10532:
-

 Summary: Capacity Scheduler Auto Queue Creation: Allow auto delete 
queue when queue is not being used
 Key: YARN-10532
 URL: https://issues.apache.org/jira/browse/YARN-10532
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Wangda Tan


It's better if we can delete auto-created queues when they are not in use for a 
period of time (like 5 mins). It will be helpful when we have a large number of 
auto-created queues (e.g. from 500 users), but only a small subset of queues 
are actively used.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue

[Wangda Tan (Jira)]

Wangda Tan created YARN-10531:
-

 Summary: Be able to disable user limit factor for 
CapacityScheduler Leaf Queue
 Key: YARN-10531
 URL: https://issues.apache.org/jira/browse/YARN-10531
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Wangda Tan


User limit factor is used to define max cap of how much resource can be 
consumed by single user. 

Under Auto Queue Creation context, it doesn't make much sense to set user limit 
factor, because initially every queue will set weight to 1.0, we want user can 
consume more resource if possible. It is hard to pre-determine how to set up 
user limit factor. So it makes more sense to add a new value (like -1) to 
indicate we will disable user limit factor 

Logic need to be changed is below: 

(Inside LeafQueue.java)

{code}
Resource maxUserLimit = Resources.none();
if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) {
  maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity,
  getUserLimitFactor());
} else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY) {
  maxUserLimit = partitionResource;
}
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-10530) CapacityScheduler ResourceLimits doesn't handle node partition well

[Wangda Tan (Jira)]

Wangda Tan created YARN-10530:
-

 Summary: CapacityScheduler ResourceLimits doesn't handle node 
partition well
 Key: YARN-10530
 URL: https://issues.apache.org/jira/browse/YARN-10530
 Project: Hadoop YARN
  Issue Type: Bug
  Components: capacity scheduler, capacityscheduler
Reporter: Wangda Tan


This is a serious bug may impact all releases, I need to do further check but I 
want to log the JIRA so we will not forget:  

ResourceLimits objects are used to handle two purposes: 

1) When there's cluster resource change, for example adding new node, or 
scheduler config reinitialize. We will pass ResourceLimits to 
updateClusterResource to queues. 

2) When allocate container, we try to pass parent's available resource to child 
to make sure child's resource allocation won't violate parent's max resource. 
For example below: 

{code}
queue used  max
--
root  1020
root.a8 10
root.a.a1 2 10
root.a.a2 6 10
{code}

Even though a.a1 has 8 resources headroom (a1.max - a1.used). But we can at 
most allocate 2 resources to a1 because root.a's limit will hit first. This 
information will be passed down from parent queue to child queue during 
assignContainers call via ResourceLimits. 

However, we only pass 1 ResourceLimits from top, for queue initialize, we 
passed in: 

{code}
root.updateClusterResource(clusterResource, new ResourceLimits(
clusterResource));
{code}

And when we update cluster resource, we only considered default partition

{code}
  // Update all children
  for (CSQueue childQueue : childQueues) {
// Get ResourceLimits of child queue before assign containers
ResourceLimits childLimits = getResourceLimitsOfChild(childQueue,
clusterResource, resourceLimits,
RMNodeLabelsManager.NO_LABEL, false);
childQueue.updateClusterResource(clusterResource, childLimits);
  }
{code}

Same for allocation logic, we passed in: (Actually I found I added a TODO item 
5 years ago).

{code}
// Try to use NON_EXCLUSIVE
assignment = getRootQueue().assignContainers(getClusterResource(),
candidates,
// TODO, now we only consider limits for parent for non-labeled
// resources, should consider labeled resources as well.
new ResourceLimits(labelManager
.getResourceByLabel(RMNodeLabelsManager.NO_LABEL,
getClusterResource())),
SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY);
{code} 

The good thing is, in the assignContainers call, we calculated child limit 
based on partition
{code} 
ResourceLimits childLimits =
  getResourceLimitsOfChild(childQueue, cluster, limits,
  candidates.getPartition(), true);
{code} 

So I think now the problem is, when a named partition has more resource than 
default partition, effective min/max resource of each queue could be wrong.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-10497) Fix an issue in CapacityScheduler which fails to delete queues

[Wangda Tan (Jira)]

Wangda Tan created YARN-10497:
-

 Summary: Fix an issue in CapacityScheduler which fails to delete 
queues
 Key: YARN-10497
 URL: https://issues.apache.org/jira/browse/YARN-10497
 Project: Hadoop YARN
  Issue Type: Improvement
Reporter: Wangda Tan


We saw an exception when using queue mutation APIs:
{code:java}
2020-11-13 16:47:46,327 WARN 
org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices: 
CapacityScheduler configuration validation failed:java.io.IOException: Queue 
root.am2cmQueueSecond not found
{code}
Which comes from this code:
{code:java}
List siblingQueues = getSiblingQueues(queueToRemove,
proposedConf);
if (!siblingQueues.contains(queueName)) {
  throw new IOException("Queue " + queueToRemove + " not found");
} 
{code}
(Inside MutableCSConfigurationProvider)

If you look at the method:
{code:java}
 
  private List getSiblingQueues(String queuePath, Configuration conf) {
String parentQueue = queuePath.substring(0, queuePath.lastIndexOf('.'));
String childQueuesKey = CapacitySchedulerConfiguration.PREFIX +
parentQueue + CapacitySchedulerConfiguration.DOT +
CapacitySchedulerConfiguration.QUEUES;
return new ArrayList<>(conf.getStringCollection(childQueuesKey));
  }
{code}
And here's capacity-scheduler.xml I got
{code:java}
yarn.scheduler.capacity.root.queuesdefault, q1, 
q2
{code}
You can notice there're spaces between default, q1, a2

So conf.getStringCollection returns:
{code:java}
default
q1
...
{code}
Which causes match issue when we try to delete the queue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-10496) [Umbrella] Support Flexible Auto Queue Creation in Capacity Scheduler

[Wangda Tan (Jira)]

Wangda Tan created YARN-10496:
-

 Summary: [Umbrella] Support Flexible Auto Queue Creation in 
Capacity Scheduler
 Key: YARN-10496
 URL: https://issues.apache.org/jira/browse/YARN-10496
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: capacity scheduler
Reporter: Wangda Tan


CapacityScheduler today doesn’t support an auto queue creation which is 
flexible enough. The current constraints: 
 * Only leaf queues can be auto-created
 * A parent can only have either static queues or dynamic ones. This causes 
multiple constraints. For example:

 * It isn’t possible to have a VIP user like Alice with a static queue 
root.user.alice with 50% capacity while the other user queues (under root.user) 
are created dynamically and they share the remaining 50% of resources.

 
 * In comparison, FairScheduler allows the following scenarios, Capacity 
Scheduler doesn’t:
 ** This implies that there is no possibility to have both dynamically created 
and static queues at the same time under root
 * A new queue needs to be created under an existing parent, while the parent 
already has static queues
 * Nested queue mapping policy, like in the following example: 

|

|
 * Here two levels of queues may need to be created 

If an application belongs to user _alice_ (who has the primary_group of 
_engineering_), the scheduler checks whether _root.engineering_ exists, if it 
doesn’t,  it’ll be created. Then scheduler checks whether 
_root.engineering.alice_ exists, and creates it if it doesn't.

 

When we try to move users from FairScheduler to CapacityScheduler, we face 
feature gaps which blocks users migrate from FS to CS.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Moving Ozone to a separated Apache project

[Wangda Tan]

+1,

Thanks,
Wangda Tan

On Tue, Sep 29, 2020 at 10:10 AM Aravindan Vijayan
 wrote:

> +1, thank you Marton.
>
> On Tue, Sep 29, 2020 at 9:17 AM Bharat Viswanadham 
> wrote:
>
> > +1
> > Thank You @Elek, Marton  for driving this.
> >
> >
> > Thanks,
> > Bharat
> >
> >
> > On Mon, Sep 28, 2020 at 10:54 AM Vivek Ratnavel <
> vivekratna...@apache.org>
> > wrote:
> >
> > > +1 for moving Ozone to a separated Top-Level Apache Project.
> > >
> > > Thanks,
> > > Vivek Subramanian
> > >
> > > On Mon, Sep 28, 2020 at 8:30 AM Hanisha Koneru
> > > 
> > > wrote:
> > >
> > > > +1
> > > >
> > > > Thanks,
> > > > Hanisha
> > > >
> > > > > On Sep 27, 2020, at 11:48 PM, Akira Ajisaka 
> > > wrote:
> > > > >
> > > > > +1
> > > > >
> > > > > Thanks,
> > > > > Akira
> > > > >
> > > > > On Fri, Sep 25, 2020 at 3:00 PM Elek, Marton  >  > > > e...@apache.org>> wrote:
> > > > >>
> > > > >> Hi all,
> > > > >>
> > > > >> Thank you for all the feedback and requests,
> > > > >>
> > > > >> As we discussed in the previous thread(s) [1], Ozone is proposed
> to
> > > be a
> > > > >> separated Apache Top Level Project (TLP)
> > > > >>
> > > > >> The proposal with all the details, motivation and history is here:
> > > > >>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Hadoop+subproject+to+Apache+TLP+proposal
> > > > >>
> > > > >> This voting runs for 7 days and will be concluded at 2nd of
> October,
> > > 6AM
> > > > >> GMT.
> > > > >>
> > > > >> Thanks,
> > > > >> Marton Elek
> > > > >>
> > > > >> [1]:
> > > > >>
> > > >
> > >
> >
> https://lists.apache.org/thread.html/rc6c79463330b3e993e24a564c6817aca1d290f186a1206c43ff0436a%40%3Chdfs-dev.hadoop.apache.org%3E
> > > > >>
> > > > >>
> > -
> > > > >> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > >  > > > yarn-dev-unsubscr...@hadoop.apache.org>
> > > > >> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> > > > <mailto:yarn-dev-h...@hadoop.apache.org>
> > > > >>
> > > > >
> > > > >
> -
> > > > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > > > <mailto:common-dev-unsubscr...@hadoop.apache.org>
> > > > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> > > > <mailto:common-dev-h...@hadoop.apache.org>
> > > >
> > >
> >
>
>
> --
> Thanks & Regards,
> Aravindan
>

[jira] [Created] (YARN-10380) Import logic of multi-node allocation in CapacityScheduler

[Wangda Tan (Jira)]

Wangda Tan created YARN-10380:
-

 Summary: Import logic of multi-node allocation in CapacityScheduler
 Key: YARN-10380
 URL: https://issues.apache.org/jira/browse/YARN-10380
 Project: Hadoop YARN
  Issue Type: Improvement
Reporter: Wangda Tan


*1) Entry point:* 
When we do multi-node allocation, we're using the same logic of async 
scheduling:
{code:java}
// Allocate containers of node [start, end)
 for (FiCaSchedulerNode node : nodes) {
  if (current++ >= start) {
     if (shouldSkipNodeSchedule(node, cs, printSkipedNodeLogging)) {
        continue;
     }
     cs.allocateContainersToNode(node.getNodeID(), false);
  }
 } {code}
Is it the most effective way to do multi-node scheduling? Should we allocate 
based on partitions? In above logic, if we have thousands of node in one 
partition, we will repeatly access all nodes of the partition thousands of 
times.

I would suggest looking at making entry-point for node-heartbeat, 
async-scheduling (single node), and async-scheduling (multi-node) to be 
different.

Node-heartbeat and async-scheduling (single node) can be still similar and 
share most of the code. 

async-scheduling (multi-node): should iterate partition first, using pseudo 
code like: 
{code:java}
for (partition : all partitions) {
  allocateContainersOnMultiNodes(getCandidate(partition))
} {code}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [ANNOUNCE] New Apache Hadoop Committer - He Xiaoqiao

[Wangda Tan]

Congrats!

Best,
Wangda Tan

On Thu, Jun 11, 2020 at 5:55 PM Tao Yang  wrote:

> Congratulations Xiaoqiao!
>
> Best,
> Tao
>
>
> --
> 发件人：Weiwei Yang 
> 发送时间：2020年6月12日(星期五) 02:24
> 收件人：Sree Vaddi 
> 抄 送：Ayush Saxena ; Chao Sun ;
> Wei-Chiu Chuang ; Hadoop Common <
> common-...@hadoop.apache.org>; Hdfs-dev ;
> yarn-dev ; mapreduce-dev <
> mapreduce-...@hadoop.apache.org>; ozone-dev 
> 主 题：Re: [ANNOUNCE] New Apache Hadoop Committer - He Xiaoqiao
>
> Congratulations Xiaoqiao!
>
> On Thu, Jun 11, 2020 at 11:20 AM Sree Vaddi  .invalid>
> wrote:
>
> > Congratulations, He Xiaoqiao.
> >
> > Thank you./Sree
> >
> >
> >
> > On Thursday, June 11, 2020, 9:54:32 AM PDT, Chao Sun <
> > sunc...@apache.org> wrote:
> >
> >  Congrats Xiaoqiao!
> >
> > On Thu, Jun 11, 2020 at 9:36 AM Ayush Saxena  wrote:
> >
> > > Congratulations He Xiaoqiao!!!
> > >
> > > -Ayush
> > >
> > > > On 11-Jun-2020, at 9:30 PM, Wei-Chiu Chuang 
> > wrote:
> > > >
> > > >  In bcc: general@
> > > >
> > > > It's my pleasure to announce that He Xiaoqiao has been elected as a
> > > > committer on the Apache Hadoop project recognizing his continued
> > > > contributions to the
> > > > project.
> > > >
> > > > Please join me in congratulating him.
> > > >
> > > > Hearty Congratulations & Welcome aboard Xiaoqiao!
> > > >
> > > > Wei-Chiu Chuang
> > > > (On behalf of the Hadoop PMC)
> > >
> > > -
> > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> > >
> > >
>
>

Re: [DISCUSS] making Ozone a separate Apache project

[Wangda Tan]

+1,

Thanks,
Wangda

On Thu, May 14, 2020 at 7:01 AM Tsuyoshi Ozawa  wrote:

> +1
>
> Thanks
> - Tsuyoshi
>
> On Thu, May 14, 2020 at 19:42 Rakesh Radhakrishnan <
> rakeshr.oz...@gmail.com>
> wrote:
>
> > +1
> >
> > Thanks,
> > Rakesh
> >
> > On Wed, May 13, 2020 at 1:23 PM Elek, Marton  wrote:
> >
> > >
> > >
> > > I would like to start a discussion to make a separate Apache project
> for
> > > Ozone
> > >
> > >
> > >
> > > ### HISTORY [1]
> > >
> > >   * Apache Hadoop Ozone development started on a feature branch of
> > > Hadoop repository (HDFS-7240)
> > >
> > >   * In the October of 2017 a discussion has been started to merge it to
> > > the Hadoop main branch
> > >
> > >   * After a long discussion it's merged to Hadoop trunk at the March of
> > > 2018
> > >
> > >   * During the discussion of the merge, it was suggested multiple times
> > > to create a separated project for the Ozone. But at that time:
> > >  1). Ozone was tightly integrated with Hadoop/HDFS
> > >  2). There was an active plan to use Block layer of Ozone (HDDS or
> > > HDSL at that time) as the block level of HDFS
> > >  3). The community of Ozone was a subset of the HDFS community
> > >
> > >   * The first beta release of Ozone was just released. Seems to be a
> > > good time before the first GA to make a decision about the future.
> > >
> > >
> > >
> > > ### WHAT HAS BEEN CHANGED
> > >
> > >   During the last years Ozone became more and more independent both at
> > > the community and code side. The separation has been suggested again
> and
> > > again (for example by Owen [2] and Vinod [3])
> > >
> > >
> > >
> > >   From COMMUNITY point of view:
> > >
> > >
> > >* Fortunately more and more new contributors are helping Ozone.
> > > Originally the Ozone community was a subset of HDFS project. But now a
> > > bigger and bigger part of the community is related to Ozone only.
> > >
> > >* It seems to be easier to _build_ the community as a separated
> > project.
> > >
> > >* A new, younger project might have different practices
> > > (communication, commiter criteria, development style) compared to old,
> > > mature project
> > >
> > >* It's easier to communicate (and improve) these standards in a
> > > separated projects with clean boundaries
> > >
> > >* Separated project/brand can help to increase the adoption rate and
> > > attract more individual contributor (AFAIK it has been seen in
> Submarine
> > > after a similar move)
> > >
> > >   * Contribution process can be communicated more easily, we can make
> > > first time contribution more easy
> > >
> > >
> > >
> > >   From CODE point of view Ozone became more and more independent:
> > >
> > >
> > >   * Ozone has different release cycle
> > >
> > >   * Code is already separated from Hadoop code base
> > > (apache/hadoop-ozone.git)
> > >
> > >   * It has separated CI (github actions)
> > >
> > >   * Ozone uses different (more strict) coding style (zero toleration of
> > > unit test / checkstyle errors)
> > >
> > >   * The code itself became more and more independent from Hadoop on
> > > Maven level. Originally it was compiled together with the in-tree
> latest
> > > Hadoop snapshot. Now it depends on released Hadoop artifacts (RPC,
> > > Configuration...)
> > >
> > >   * It starts to use multiple version of Hadoop (on client side)
> > >
> > >   * Volume of resolved issues are already very high on Ozone side
> (Ozone
> > > had slightly more resolved issues than HDFS/YARN/MAPREDUCE/COMMON all
> > > together in the last 2-3 months)
> > >
> > >
> > > Summary: Before the first Ozone GA release, It seems to be a good time
> > > to discuss the long-term future of Ozone. Managing it as a separated
> TLP
> > > project seems to have more benefits.
> > >
> > >
> > > Please let me know what your opinion is...
> > >
> > > Thanks a lot,
> > > Marton
> > >
> > >
> > >
> > >
> > >
> > > [1]: For more details, see:
> > > https://github.com/apache/hadoop-ozone/blob/master/HISTORY.md
> > >
> > > [2]:
> > >
> > >
> >
> https://lists.apache.org/thread.html/0d0253f6e5fa4f609bd9b917df8e1e4d8848e2b7fdb3099b730095e6%40%3Cprivate.hadoop.apache.org%3E
> > >
> > > [3]:
> > >
> > >
> >
> https://lists.apache.org/thread.html/8be74421ea495a62e159f2b15d74627c63ea1f67a2464fa02c85d4aa%40%3Chdfs-dev.hadoop.apache.org%3E
> > >
> > > -
> > > To unsubscribe, e-mail: ozone-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: ozone-dev-h...@hadoop.apache.org
> > >
> > >
> >
>

[jira] [Resolved] (YARN-10151) Disable Capacity Scheduler's move app between queue functionality

[Wangda Tan (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-10151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan resolved YARN-10151.
---
Resolution: Won't Fix

Thanks folks for commenting about YARN-9838. I think we don't need this change 
now given we have a fix of the reported issue already.

> Disable Capacity Scheduler's move app between queue functionality
> -
>
> Key: YARN-10151
> URL: https://issues.apache.org/jira/browse/YARN-10151
> Project: Hadoop YARN
>  Issue Type: Bug
>    Reporter: Wangda Tan
>Priority: Critical
>
> Saw this happened in many clusters: Capacity Scheduler cannot work correctly 
> with the move app between queue features. It will cause weird JMX issue, 
> resource accounting issue, etc. In a lot of causes it will cause RM 
> completely hung and available resource became negative, nothing can be 
> allocated after that. We should turn off CapacityScheduler's move app between 
> queue feature. (see: 
> {{org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler#moveApplication}}
>  )



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] EOL Hadoop branch-2.8

[Wangda Tan]

+1

On Mon, Mar 2, 2020 at 8:50 PM Akira Ajisaka  wrote:

> +1
>
> -Akira
>
> On Tue, Mar 3, 2020 at 4:55 AM Ayush Saxena  wrote:
>
> > +1 for marking 2.8 EOL
> >
> > -Ayush
> >
> > > On 03-Mar-2020, at 12:18 AM, Wei-Chiu Chuang 
> wrote:
> > >
> > > I am sorry I forgot to start a VOTE thread.
> > >
> > > This is the "official" vote thread to mark branch-2.8 End of Life. This
> > is
> > > based on the following thread and the tracking jira (HADOOP-16880
> > > ).
> > >
> > > This vote will run for 7 days and conclude on March 9th (Mon) 11am PST.
> > >
> > > Please feel free to share your thoughts.
> > >
> > > Thanks,
> > > Weichiu
> > >
> > >> On Mon, Feb 24, 2020 at 10:28 AM Wei-Chiu Chuang <
> weic...@cloudera.com>
> > >> wrote:
> > >>
> > >> Looking at the EOL policy wiki:
> > >>
> >
> https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches
> > >>
> > >> The Hadoop community can still elect to make security update for
> EOL'ed
> > >> releases.
> > >>
> > >> I think the EOL is to give more clarity to downstream applications
> (such
> > >> as HBase) the guidance of which Hadoop release lines are still active.
> > >> Additionally, I don't think it is sustainable to maintain 6 concurrent
> > >> release lines in this big project, which is why I wanted to start this
> > >> discussion.
> > >>
> > >> Thoughts?
> > >>
> > >>> On Mon, Feb 24, 2020 at 10:22 AM Sunil Govindan 
> > wrote:
> > >>>
> > >>> Hi Wei-Chiu
> > >>>
> > >>> Extremely sorry for the late reply here.
> > >>> Cud u pls help to add more clarity on defining what will happen for
> > >>> branch-2.8 when we call EOL.
> > >>> Does this mean that, no more release coming out from this branch, or
> > some
> > >>> more additional guidelines?
> > >>>
> > >>> - Sunil
> > >>>
> > >>>
> > >>> On Mon, Feb 24, 2020 at 11:47 PM Wei-Chiu Chuang
> > >>>  wrote:
> > >>>
> >  This thread has been running for 7 days and no -1.
> > 
> >  Don't think we've established a formal EOL process, but to publicize
> > the
> >  EOL, I am going to file a jira, update the wiki and post the
> > >>> announcement
> >  to general@ and user@
> > 
> >  On Wed, Feb 19, 2020 at 1:40 PM Dinesh Chitlangia <
> > >>> dineshc@gmail.com>
> >  wrote:
> > 
> > > Thanks Wei-Chiu for initiating this.
> > >
> > > +1 for 2.8 EOL.
> > >
> > > On Tue, Feb 18, 2020 at 10:48 PM Akira Ajisaka <
> aajis...@apache.org>
> > > wrote:
> > >
> > >> Thanks Wei-Chiu for starting the discussion,
> > >>
> > >> +1 for the EoL.
> > >>
> > >> -Akira
> > >>
> > >> On Tue, Feb 18, 2020 at 4:59 PM Ayush Saxena 
> >  wrote:
> > >>
> > >>> Thanx Wei-Chiu for initiating this
> > >>> +1 for marking 2.8 EOL
> > >>>
> > >>> -Ayush
> > >>>
> >  On 17-Feb-2020, at 11:14 PM, Wei-Chiu Chuang <
> > >>> weic...@apache.org>
> > >> wrote:
> > 
> >  The last Hadoop 2.8.x release, 2.8.5, was GA on September 15th
> >  2018.
> > 
> >  It's been 17 months since the release and the community by and
> >  large
> > >> have
> >  moved up to 2.9/2.10/3.x.
> > 
> >  With Hadoop 3.3.0 over the horizon, is it time to start the EOL
> > >>> discussion
> >  and reduce the number of active branches?
> > >>>
> > >>>
> > >>> -
> > >>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > >>> For additional commands, e-mail:
> > >>> common-dev-h...@hadoop.apache.org
> > >>>
> > >>>
> > >>
> > >
> > 
> > >>>
> > >>
> >
> > -
> > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
> >
> >
>

[jira] [Created] (YARN-10170) Should revisit mix-usage of percentage-based and absolute-value-based min/max resource in CapacityScheduler

[Wangda Tan (Jira)]

Wangda Tan created YARN-10170:
-

 Summary: Should revisit mix-usage of percentage-based and 
absolute-value-based min/max resource in CapacityScheduler
 Key: YARN-10170
 URL: https://issues.apache.org/jira/browse/YARN-10170
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Wangda Tan


This should be finished after YARN-10169. (If we can get this one easily, we 
should do this one instead of YARN-10169).

Absolute resource means mem=x, vcore=y.

Percentage resource means x%

We should not allow percentage-based child, but absolute-based parent. (root is 
considered as percentage-based).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-10167) Need validate c-s.xml after converting

[Wangda Tan (Jira)]

Wangda Tan created YARN-10167:
-

 Summary: Need validate c-s.xml after converting
 Key: YARN-10167
 URL: https://issues.apache.org/jira/browse/YARN-10167
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Wangda Tan


Currently we just generated c-s.xml, but we didn't validate that. To make sure 
the c-s.xml is correct after conversion, it's better to initialize the CS 
scheduler using configs.

Also, in the test, we should try to leverage MockRM to validate generated 
configs as much as we could.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-10151) Disable Capacity Scheduler's move app between queue functionality

[Wangda Tan (Jira)]

Wangda Tan created YARN-10151:
-

 Summary: Disable Capacity Scheduler's move app between queue 
functionality
 Key: YARN-10151
 URL: https://issues.apache.org/jira/browse/YARN-10151
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Wangda Tan


Saw this happened in many clusters: Capacity Scheduler cannot work correctly 
with the move app between queue features. It will cause weird JMX issue, 
resource accounting issue, etc. In a lot of causes it will cause RM completely 
hung and available resource became negative, nothing can be allocated after 
that. We should turn off CapacityScheduler's move app between queue feature. 
(see: 
{{org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler#moveApplication}}
 )



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] Hadoop 2019 Release Planning

[Wangda Tan]

Hi guys,

Previously we are using
https://cwiki.apache.org/confluence/display/HADOOP/Roadmap for 3.1 and 3.2.
So to reduce confusing, I suggest to use one of the
https://cwiki.apache.org/confluence/display/HADOOP/Roadmap or
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Active+Release+Lines
for the release plans.

Thanks,
Wangda

On Wed, Jan 8, 2020 at 5:56 AM Ayush Saxena  wrote:

> Hi Gabor
>
> Seams like you need to create one, I too can't find anything related.
>
> Most I could find was :
>
> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.1.3
>
> -Ayush
>
>
> On Wed, 8 Jan 2020 at 18:45, Gabor Bota 
> wrote:
>
> > I'm volunteering to be the release manager for 3.1.4.
> > Where can I find a wiki page for the 3.1 line? I can not find one here:
> >
> >
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Active+Release+Lines
> >
> > On Tue, Jan 7, 2020 at 3:41 PM Steve Loughran
>  > >
> > wrote:
> >
> > > I'm thinking of doing a backport of most of the hadoop-aws changes to
> > > branch-3.2, for the next 3.2.x release; they are all self contained and
> > > will benefit many (they will need to cope with the older mockito
> version,
> > > but I have to deal with that in-house already).
> > >
> > > one change is the new openFile() builder API. I'd like to wrap that up
> > with
> > > a little improvement
> https://issues.apache.org/jira/browse/HADOOP-16759;
> > > That way for all releases with the API, it's consistent.
> > >
> > > (that withStatus() feature gives extra performance and ensures that
> > > etag/version can be used to get the explicit version you want.)
> > >
> > > On Tue, Jan 7, 2020 at 2:18 AM Akira Ajisaka 
> > wrote:
> > >
> > > > >  I am interested on 3.3 release ..will act as RM .will update the
> > wiki
> > > as
> > > > well..
> > > >
> > > > Thanks Brahma for your reply. I'll help you as co-RM.
> > > > We will send announcements (cutting branches, code freeze, and so on)
> > in
> > > > another thread.
> > > >
> > > > Thanks,
> > > > Akira
> > > >
> > > > On Tue, Jan 7, 2020 at 4:32 AM Wangda Tan 
> wrote:
> > > >
> > > > > Hi guys,
> > > > >
> > > > > Thanks for the update and for volunteering to be RM.
> > > > >
> > > > > I just did a quick check:
> > > > > 3.1.4 has 52 patches resolved. (3.1.3 Released on Oct 21)
> > > > > 3.2.2 has 46 patches resolved. (3.2.1 Released on Sep 22)
> > > > > 3.3.0 has .. many patches sitting here so we definitely need a
> > release.
> > > > >
> > > > > If Akira and Brahma you guys can be co-RMs for 3.3.0 that would be
> > > great.
> > > > >
> > > > > Hadoop 3.2.1 is released on Sep 22 which is 3+ months ago, and I
> saw
> > > > > community started to have large prod deployment on 3.2.x, Gabor if
> > you
> > > > have
> > > > > bandwidth to help releases, I think we can do 3.2.2 first then
> 3.1.4.
> > > > >
> > > > > Thoughts?
> > > > > - Wangda
> > > > >
> > > > > On Mon, Jan 6, 2020 at 5:50 AM Brahma Reddy Battula <
> > bra...@apache.org
> > > >
> > > > > wrote:
> > > > >
> > > > >> Thanks Akira for resuming this..
> > > > >>
> > > > >>  I am interested on 3.3 release ..will act as RM .will update the
> > wiki
> > > > as
> > > > >> well..
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Mon, 6 Jan 2020 at 6:08 PM, Gabor Bota <
> gabor.b...@cloudera.com
> > > > .invalid>
> > > > >> wrote:
> > > > >>
> > > > >>> I'm interested in doing a release of hadoop.
> > > > >>> The version we need an RM is 3.1.3 right? What's the target date
> > for
> > > > >>> that?
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Gabor
> > > > >>>
> > > > >>> On Mon, Jan 6, 2020 at 8:31 AM Akira Ajisaka <
> aajis...@apache.org>
> > > > >>> wrote:
> > > > >>>
> > > > >>> > Thank you Wangda.
> > > &

Re: [DISCUSS] Hadoop 2019 Release Planning

[Wangda Tan]

Hi guys,

Thanks for the update and for volunteering to be RM.

I just did a quick check:
3.1.4 has 52 patches resolved. (3.1.3 Released on Oct 21)
3.2.2 has 46 patches resolved. (3.2.1 Released on Sep 22)
3.3.0 has .. many patches sitting here so we definitely need a release.

If Akira and Brahma you guys can be co-RMs for 3.3.0 that would be great.

Hadoop 3.2.1 is released on Sep 22 which is 3+ months ago, and I saw
community started to have large prod deployment on 3.2.x, Gabor if you have
bandwidth to help releases, I think we can do 3.2.2 first then 3.1.4.

Thoughts?
- Wangda

On Mon, Jan 6, 2020 at 5:50 AM Brahma Reddy Battula 
wrote:

> Thanks Akira for resuming this..
>
>  I am interested on 3.3 release ..will act as RM .will update the wiki as
> well..
>
>
>
> On Mon, 6 Jan 2020 at 6:08 PM, Gabor Bota 
> wrote:
>
>> I'm interested in doing a release of hadoop.
>> The version we need an RM is 3.1.3 right? What's the target date for that?
>>
>> Thanks,
>> Gabor
>>
>> On Mon, Jan 6, 2020 at 8:31 AM Akira Ajisaka  wrote:
>>
>> > Thank you Wangda.
>> >
>> > Now it's 2020. Let's release Hadoop 3.3.0.
>> > I created a wiki page for tracking blocker/critical issues for 3.3.0 and
>> > I'll check the issues in the list.
>> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.3+Release
>> > If you find blocker/critical issues in trunk, please set the target
>> version
>> > to 3.3.0 for tracking.
>> >
>> > > We still need RM for 3.3.0 and 3.1.3.
>> > I can work as a release manager for 3.3.0. Is there anyone who wants to
>> be
>> > a RM?
>> >
>> > Thanks and regards,
>> > Akira
>> >
>> > On Fri, Aug 16, 2019 at 9:28 PM zhankun tang 
>> > wrote:
>> >
>> > > Thanks Wangda for bring this up!
>> > >
>> > > I ran the submarine 0.2.0 release before with a lot of help from folks
>> > > especially Sunil. :D
>> > > And this time I would like to help to release the 3.1.4. Thanks!
>> > >
>> > > BR,
>> > > Zhankun
>> > >
>> > > Hui Fei 于2019年8月16日 周五下午7:19写道：
>> > >
>> > > > Hi Wangda,
>> > > > Thanks for bringing this up!
>> > > > Looking forward to see HDFS 3.x is widely used，but RollingUpgrade
>> is a
>> > > > problem.
>> > > > Hope commiters watch and review these issues, Thanks
>> > > > https://issues.apache.org/jira/browse/HDFS-13596
>> > > > https://issues.apache.org/jira/browse/HDFS-14396
>> > > >
>> > > > Wangda Tan  于2019年8月10日周六 上午10:59写道：
>> > > >
>> > > > > Hi all,
>> > > > >
>> > > > > Hope this email finds you well
>> > > > >
>> > > > > I want to hear your thoughts about what should be the release plan
>> > for
>> > > > > 2019.
>> > > > >
>> > > > > In 2018, we released:
>> > > > > - 1 maintenance release of 2.6
>> > > > > - 3 maintenance releases of 2.7
>> > > > > - 3 maintenance releases of 2.8
>> > > > > - 3 releases of 2.9
>> > > > > - 4 releases of 3.0
>> > > > > - 2 releases of 3.1
>> > > > >
>> > > > > Total 16 releases in 2018.
>> > > > >
>> > > > > In 2019, by far we only have two releases:
>> > > > > - 1 maintenance release of 3.1
>> > > > > - 1 minor release of 3.2.
>> > > > >
>> > > > > However, the community put a lot of efforts to stabilize features
>> of
>> > > > > various release branches.
>> > > > > There're:
>> > > > > - 217 fixed patches in 3.1.3 [1]
>> > > > > - 388 fixed patches in 3.2.1 [2]
>> > > > > - 1172 fixed patches in 3.3.0 [3] (OMG!)
>> > > > >
>> > > > > I think it is the time to do maintenance releases of 3.1/3.2 and
>> do a
>> > > > minor
>> > > > > release for 3.3.0.
>> > > > >
>> > > > > In addition, I saw community discussion to do a 2.8.6 release for
>> > > > security
>> > > > > fixes.
>> > > > >
>> > > > > Any other releases? I think there're release plans for Ozone as
>> well.
>> > > And
>> > > > > please add your thoughts.
>> > > > >
>> > > > > Volunteers welcome! If you have interests to run a release as
>> Release
>> > > > > Manager (or co-Resource Manager), please respond to this email
>> thread
>> > > so
>> > > > we
>> > > > > can coordinate.
>> > > > >
>> > > > > Thanks,
>> > > > > Wangda Tan
>> > > > >
>> > > > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution =
>> Fixed
>> > > AND
>> > > > > fixVersion = 3.1.3
>> > > > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution =
>> Fixed
>> > > AND
>> > > > > fixVersion = 3.2.1
>> > > > > [3] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution =
>> Fixed
>> > > AND
>> > > > > fixVersion = 3.3.0
>> > > > >
>> > > >
>> > >
>> >
>>
> --
>
>
>
> --Brahma Reddy Battula
>

Re: [DISCUSS] Making 2.10 the last minor 2.x release

[Wangda Tan]

+1, thanks Jonathan for bringing this up!

On Fri, Nov 15, 2019 at 11:41 AM epa...@apache.org 
wrote:

> Thanks Jonathan for opening the discussion.
>
> I am not in favor of this proposal. 2.10 was very recently released, and
> moving to 2.10 will take some time for the community. It seems premature to
> make a decision at this point that there will never be a need for a 2.11
> release.
>
> -Eric
>
>
>  On Thursday, November 14, 2019, 8:51:59 PM CST, Jonathan Hung <
> jyhung2...@gmail.com> wrote:
>
> Hi folks,
>
> Given the release of 2.10.0, and the fact that it's intended to be a bridge
> release to Hadoop 3.x [1], I'm proposing we make 2.10.x the last minor
> release line in branch-2. Currently, the main issue is that there's many
> fixes going into branch-2 (the theoretical 2.11.0) that's not going into
> branch-2.10 (which will become 2.10.1), so the fixes in branch-2 will
> likely never see the light of day unless they are backported to
> branch-2.10.
>
> To do this, I propose we:
>
>   - Delete branch-2.10
>   - Rename branch-2 to branch-2.10
>   - Set version in the new branch-2.10 to 2.10.1-SNAPSHOT
>
> This way we get all the current branch-2 fixes into the 2.10.x release
> line. Then the commit chain will look like: trunk -> branch-3.2 ->
> branch-3.1 -> branch-2.10 -> branch-2.9 -> branch-2.8
>
> Thoughts?
>
> Jonathan Hung
>
> [1] https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg29479.html
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [DISCUSS] Remove Ozone and Submarine from Hadoop repo

[Wangda Tan]

+1, we should do that earlier than later. It is a code change that I think
we don't need to wait for 7 days vote.

On Thu, Oct 24, 2019 at 4:18 PM Anu Engineer 
wrote:

> +1 for Ozone. We are in our own repo now. It would be good to remove this
> code from Hadoop, otherwise it will confuse new contributors.
> I would like to add a git tag tro Hadoop, so that people have the ability
> to sync back and see the code evolution.
>
> --Anu
>
> On Thu, Oct 24, 2019 at 4:03 PM Giovanni Matteo Fumarola <
> giovanni.fumar...@gmail.com> wrote:
>
> > +1
> >
> > Thanks Wei-Chiu for creating HADOOP-16670.
> >
> > On Thu, Oct 24, 2019 at 12:56 PM Wei-Chiu Chuang 
> > wrote:
> >
> > > +1 filed HADOOP-16670 <
> > https://issues.apache.org/jira/browse/HADOOP-16670>
> > > for
> > > stripping the Submarine code.
> > >
> > > On Thu, Oct 24, 2019 at 12:14 PM Subru Krishnan 
> > wrote:
> > >
> > > > +1.
> > > >
> > > > Thanks,
> > > > Subru
> > > >
> > > > On Thu, Oct 24, 2019 at 12:51 AM 张铎(Duo Zhang) <
> palomino...@gmail.com>
> > > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > Akira Ajisaka  于2019年10月24日周四 下午3:21写道：
> > > > >
> > > > > > Hi folks,
> > > > > >
> > > > > > Both Ozone and Apache Submarine have separate repositories.
> > > > > > Can we remove these modules from hadoop-trunk?
> > > > > >
> > > > > > Regards,
> > > > > > Akira
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Submarine spin-off proposal approved by Apache Board

[Wangda Tan]

Hi folks,

Name search is approved by TRADEMARK yesterday so now we will continue the
spin-off process.

I just created
https://cwiki.apache.org/confluence/display/HADOOP/Submarine+Spin-Off+Plan
which is items I can think about, please add items which I missed.

Thanks,
Wangda

On Wed, Oct 23, 2019 at 4:08 AM Vinod Kumar Vavilapalli 
wrote:

> Is this plan already tracked somewhere?
>
> Thanks
> +Vinod
>
> On Oct 17, 2019, at 4:51 PM, Vinod Kumar Vavilapalli 
> wrote:
>
> On the next steps, we also need to have a plan sketched out on what we do
> with the code changes and releases under the Hadoop repositories. It’d make
> sense to create a wiki page with a list of action items for this transition.
>
>
>

Re: [Discuss] Hadoop-Ozone repository mailing list configurations

[Wangda Tan]

We're going to fix the Submarine email list issues once spin-off works
start

On Wed, Oct 23, 2019 at 2:39 PM Matt Foley 
wrote:

> Definitely yes on ‘ozone-issues’.  Whether we want to keep ozone-dev and
> hdfs-dev together or separate, I’m neutral.
> Thanks,
> —Matt
>
> On Oct 23, 2019, at 2:11 PM, Elek, Marton  wrote:
>
> Thanks to report this problem Rohith,
>
> Yes, it seems to be configured with the wrong mailing list.
>
> I think the right fix is to create ozone-dev@ and ozone-issues@ and use
> them instead of hdfs-(dev/issues).
>
> Is there any objections against creating new ozone-* mailing lists?
>
> Thanks,
> Marton
>
>
> On 10/21/19 6:03 AM, Rohith Sharma K S wrote:
> > + common/yarn and mapreduce/submarine
> > Looks like same issue in submarine repository also !
> > On Mon, 21 Oct 2019 at 09:30, Rohith Sharma K S <
> rohithsharm...@apache.org>
> > wrote:
> >> Folks,
> >>
> >> In Hadoop world, any mailing list has its own purposes as below
> >> 1. hdfs/common/yarn/mapreduce-*dev *mailing list is meant for developer
> >> discussion purpose.
> >> 2. hdfs/common/yarn/mapreduce*-issues* mailing list used for comments
> >> made in the issues.
> >>
> >>  It appears Hadoop-Ozone repository configured *hdfs-dev *mailing list
> >> for *hdfs-issues* list also. As a result hdfs-dev mailing list is
> >> bombarded with every comment made in hadoop-ozone repository.
> >>
> >>
> >> Could it be fixed?
> >>
> >> -Rohith Sharma K S
> >>
> >>
> >>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>
>

Submarine spin-off proposal approved by Apache Board

[Wangda Tan]

Hi Devs,

I'm happy to let everybody know that Submarine spin-off proposal finally
got approved by Apache board. We will work with Apache board to finish the
rest of spin-off processes.

Where we're now?
- We need to finish the name search to make sure "Apache Submarine" is a
good name. I have done search works and filed
https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-171 and will wait
for trademark members to approve the proposal.
- After name approved by trademark (no ETAs yet), we're going to work with
Apache Infra to create resources like email lists, git repo, etc. This also
mean that we need to use existing email lists, JIRAs, Git to do our
day-to-day discussion/development, nothing will be changed.
- After all the required resources created, we will start to move code,
website, etc. to the new location.

Thanks for all the from so many people during the long journey, let's
continue to drive our Submarine to deeper and deeper!

Please let me know if you have any questions.

Thanks,
Wangda

[NEED HELP] Hadoop 3.x Production Deployment Can be Publicly Talk About?

[Wangda Tan]

Hi devs and users,

Tomorrow (sorry for the short notice) we will do a presentation at Strata
Data Conf @ NY for a Community update of Hadoop 3.x. I'm thinking to create
a slide about existing production deployment on Hadoop 3.x. Basically, I
want to put a logo wall with a list of big names so we can encourage more
users to upgrade to 3.x.

I knew there are tons of users are on 3.x already, but only a few of them
have public slides, I don't get a permission to put other non-public use
cases to the slide. So if you are:
- Using 3.x in production. (Ideally large scale, using some new features or
running in new environment like on cloud, using GPU, etc.).
- When I saying 3.x it could be Apache Hadoop 3.x, HDP 3.x, CDH 6.x or any
distribution with Apache Hadoop 3.x as base.

Please respond your company name w/ logo (it it is not too obvious) to this
email (either public or private to me). If you could include a short
summary (which can be publicly shared) of the use cases that will be very
helpful.

If the number of companies responded is too low, I may skip putting a logo
wall.

Hope to get your feedbacks soon.

Thanks,
Wangda Tan

Re: [NOTICE] Building trunk needs protoc 3.7.1

[Wangda Tan]

Hi Vinay,

Thanks for the clarification.

Do you have a timeline about all you described works w.r.t.  the
compatibility will be completed? I'm asking this is because we need to
release 3.3.0 earlier if possible since there're 1k+ of patches in 3.3.0
already, we should get it out earlier.

If the PB work will take more time, do you think if we should create a
branch for 3.3, revert PB changes from branch-3.3, and keep on working on
PB for the next minor release? (or major release if we do see some
compatibility issue in the future).

Just my $0.02

Thanks,
Wangda

On Mon, Sep 23, 2019 at 5:43 AM Steve Loughran 
wrote:

> aah, that makes sense
>
> On Sun, Sep 22, 2019 at 6:11 PM Vinayakumar B 
> wrote:
>
> > Thanks Steve.
> >
> > Idea is not to shade all artifacts.
> > Instead maintain one artifact ( hadoop-thirdparty) which have all such
> > dependencies ( com.google.* may be), add  this artifact as dependency in
> > hadoop modules. Use shaded classes directly in the code of hadoop modules
> > instead of shading at package phase.
> >
> > Hbase, ozone and ratis already following this way. The artifact (
> > hadoop-thirdparty) with shaded dependencies can be maintained in a
> separate
> > repo as suggested by stack on HADOOP-13363 or could be maintained as a
> > separate module in Hadoop repo. If maintained in separate repo, need to
> > build this only when there are changes related to shaded dependencies.
> >
> >
> > -Vinay
> >
> > On Sun, 22 Sep 2019, 10:11 pm Steve Loughran, 
> wrote:
> >
> > >
> > >
> > > On Sun, Sep 22, 2019 at 3:22 PM Vinayakumar B  >
> > > wrote:
> > >
> > >>Protobuf provides Wire compatibility between releases.. but not
> > >> guarantees the source compatibility in generated sources. There will
> be
> > a
> > >> problem in compatibility if anyone uses generated protobuf message
> > outside
> > >> of Hadoop modules. Which ideally shouldn't be as generated sources are
> > not
> > >> public APIs.
> > >>
> > >>There should not be any compatibility problems between releases in
> > >> terms
> > >> of communication provided both uses same syntax (proto2) of proto
> > message.
> > >> This I have verified by communication between protobuf 2.5.0 client
> with
> > >> protobuf 3.7.1 server.
> > >>
> > >>To avoid the downstream transitive dependency classpath problem,
> who
> > >> might be using protobuf 2.5.0 classes, planning to shade the 3.7.1
> > classes
> > >> and its usages in all hadoop modules.. and keep 2.5.0 jar back in
> hadoop
> > >> classpath.
> > >>
> > >> Hope I have answered your question.
> > >>
> > >> -Vinay
> > >>
> > >>
> > > While I support the move and CP isolation, this is going to (finally)
> > > force us to make shaded versions of all artifacts which we publish with
> > the
> > > intent of them being loaded on the classpath of other applications
> > >
> >
>

Re: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk source tree

[Wangda Tan]

+1 (binding).

>From my experiences of Submarine project, I think moving to a separate repo
helps.

- Wangda

On Tue, Sep 17, 2019 at 11:41 AM Subru Krishnan  wrote:

> +1 (binding).
>
> IIUC, there will not be an Ozone module in trunk anymore as that was my
> only concern from the original discussion thread? IMHO, this should be the
> default approach for new modules.
>
> On Tue, Sep 17, 2019 at 9:58 AM Salvatore LaMendola (BLOOMBERG/ 731 LEX) <
> slamendo...@bloomberg.net> wrote:
>
> > +1
> >
> > From: e...@apache.org At: 09/17/19 05:48:32To:
> hdfs-...@hadoop.apache.org,
> > mapreduce-...@hadoop.apache.org,  common-...@hadoop.apache.org,
> > yarn-dev@hadoop.apache.org
> > Subject: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk
> > source tree
> >
> >
> > TLDR; I propose to move Ozone related code out from Hadoop trunk and
> > store it in a separated *Hadoop* git repository apache/hadoop-ozone.git
> >
> >
> > When Ozone was adopted as a new Hadoop subproject it was proposed[1] to
> > be part of the source tree but with separated release cadence, mainly
> > because it had the hadoop-trunk/SNAPSHOT as compile time dependency.
> >
> > During the last Ozone releases this dependency is removed to provide
> > more stable releases. Instead of using the latest trunk/SNAPSHOT build
> > from Hadoop, Ozone uses the latest stable Hadoop (3.2.0 as of now).
> >
> > As we have no more strict dependency between Hadoop trunk SNAPSHOT and
> > Ozone trunk I propose to separate the two code base from each other with
> > creating a new Hadoop git repository (apache/hadoop-ozone.git):
> >
> > With moving Ozone to a separated git repository:
> >
> >   * It would be easier to contribute and understand the build (as of now
> > we always need `-f pom.ozone.xml` as a Maven parameter)
> >   * It would be possible to adjust build process without breaking
> > Hadoop/Ozone builds.
> >   * It would be possible to use different Readme/.asf.yaml/github
> > template for the Hadoop Ozone and core Hadoop. (For example the current
> > github template [2] has a link to the contribution guideline [3]. Ozone
> > has an extended version [4] from this guideline with additional
> > information.)
> >   * Testing would be more safe as it won't be possible to change core
> > Hadoop and Hadoop Ozone in the same patch.
> >   * It would be easier to cut branches for Hadoop releases (based on the
> > original consensus, Ozone should be removed from all the release
> > branches after creating relase branches from trunk)
> >
> >
> > What do you think?
> >
> > Thanks,
> > Marton
> >
> > [1]:
> >
> >
> https://lists.apache.org/thread.html/c85e5263dcc0ca1d13cbbe3bcfb53236784a39111b8
> > c353f60582eb4@%3Chdfs-dev.hadoop.apache.org%3E
> > [2]:
> >
> >
> https://github.com/apache/hadoop/blob/trunk/.github/pull_request_template.md
> > [3]:
> https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
> > [4]:
> >
> >
> https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute+to+Ozone
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> >
> >
>

Re: [VOTE] Moving Submarine to a separate Apache project proposal

[Wangda Tan]

Thanks everyone for voting, support, and suggestions!

Best,
Wangda


On Sun, Sep 8, 2019 at 10:57 PM Sree Vaddi  wrote:

> Congratulations, Wangda. Thank you.
>
> Let's work towards it's better adoption.
>
>
> Thank you.
> /Sree
>
>
>
>
> On Sunday, September 8, 2019, 10:30:49 PM PDT, Wangda Tan <
> wheele...@gmail.com> wrote:
>
>
> Thanks everybody for voting! Let me conclude the voting thread:
>
> We got 16 binding +1s from PMCs. And 29 +1s from Committers/Contributors.
> With no veto.
>
> The vote now passed, we will work with the Apache community to get new
> project created and continue the spin-off process.
>
> Best,
> Wangda Tan
>
>
> On Sun, Sep 8, 2019 at 11:24 AM Chen Liang  wrote:
>
> > Late +1.  I am interested in the project too. Please include me.
> >
> > Regards,
> > Chen
> >
> > Varun Saxena  于2019年9月7日周六 上午10:35写道：
> >
> >> Belated +1.
> >> I too am interested in the project. Please include me.
> >>
> >> Regards,
> >> Varun Saxena
> >>
> >> On Sat, Sep 7, 2019 at 9:35 PM Sunil Govindan 
> wrote:
> >>
> >> > +1 to the proposal.
> >> > Thanks to the community for the great response.
> >> >
> >> > - Sunil
> >> >
> >> > On Sun, Sep 1, 2019 at 10:49 AM Wangda Tan 
> wrote:
> >> >
> >> > > Hi all,
> >> > >
> >> > > As we discussed in the previous thread [1],
> >> > >
> >> > > I just moved the spin-off proposal to CWIKI and completed all TODO
> >> parts.
> >> > >
> >> > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/HADOOP/Submarine+Project+Spin-Off+to+TLP+Proposal
> >> > >
> >> > > If you have interests to learn more about this. Please review the
> >> > proposal
> >> > > let me know if you have any questions/suggestions for the proposal.
> >> This
> >> > > will be sent to board post voting passed. (And please note that the
> >> > > previous voting thread [2] to move Submarine to a separate Github
> repo
> >> > is a
> >> > > necessary effort to move Submarine to a separate Apache project but
> >> not
> >> > > sufficient so I sent two separate voting thread.)
> >> > >
> >> > > Please let me know if I missed anyone in the proposal, and reply if
> >> you'd
> >> > > like to be included in the project.
> >> > >
> >> > > This voting runs for 7 days and will be concluded at Sep 7th, 11 PM
> >> PDT.
> >> > >
> >> > > Thanks,
> >> > > Wangda Tan
> >> > >
> >> > > [1]
> >> > >
> >> > >
> >> >
> >>
> https://lists.apache.org/thread.html/4a2210d567cbc05af92c12aa6283fd09b857ce209d537986ed800029@%3Cyarn-dev.hadoop.apache.org%3E
> >> > > [2]
> >> > >
> >> > >
> >> >
> >>
> https://lists.apache.org/thread.html/6e94469ca105d5a15dc63903a541bd21c7ef70b8bcff475a16b5ed73@%3Cyarn-dev.hadoop.apache.org%3E
> >> > >
> >> >
> >>
> >
>

Re: [VOTE] Moving Submarine to a separate Apache project proposal

[Wangda Tan]

Thanks everybody for voting! Let me conclude the voting thread:

We got 16 binding +1s from PMCs. And 29 +1s from Committers/Contributors.
With no veto.

The vote now passed, we will work with the Apache community to get new
project created and continue the spin-off process.

Best,
Wangda Tan


On Sun, Sep 8, 2019 at 11:24 AM Chen Liang  wrote:

> Late +1.   I am interested in the project too. Please include me.
>
> Regards,
> Chen
>
> Varun Saxena  于2019年9月7日周六 上午10:35写道：
>
>> Belated +1.
>> I too am interested in the project. Please include me.
>>
>> Regards,
>> Varun Saxena
>>
>> On Sat, Sep 7, 2019 at 9:35 PM Sunil Govindan  wrote:
>>
>> > +1 to the proposal.
>> > Thanks to the community for the great response.
>> >
>> > - Sunil
>> >
>> > On Sun, Sep 1, 2019 at 10:49 AM Wangda Tan  wrote:
>> >
>> > > Hi all,
>> > >
>> > > As we discussed in the previous thread [1],
>> > >
>> > > I just moved the spin-off proposal to CWIKI and completed all TODO
>> parts.
>> > >
>> > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/HADOOP/Submarine+Project+Spin-Off+to+TLP+Proposal
>> > >
>> > > If you have interests to learn more about this. Please review the
>> > proposal
>> > > let me know if you have any questions/suggestions for the proposal.
>> This
>> > > will be sent to board post voting passed. (And please note that the
>> > > previous voting thread [2] to move Submarine to a separate Github repo
>> > is a
>> > > necessary effort to move Submarine to a separate Apache project but
>> not
>> > > sufficient so I sent two separate voting thread.)
>> > >
>> > > Please let me know if I missed anyone in the proposal, and reply if
>> you'd
>> > > like to be included in the project.
>> > >
>> > > This voting runs for 7 days and will be concluded at Sep 7th, 11 PM
>> PDT.
>> > >
>> > > Thanks,
>> > > Wangda Tan
>> > >
>> > > [1]
>> > >
>> > >
>> >
>> https://lists.apache.org/thread.html/4a2210d567cbc05af92c12aa6283fd09b857ce209d537986ed800029@%3Cyarn-dev.hadoop.apache.org%3E
>> > > [2]
>> > >
>> > >
>> >
>> https://lists.apache.org/thread.html/6e94469ca105d5a15dc63903a541bd21c7ef70b8bcff475a16b5ed73@%3Cyarn-dev.hadoop.apache.org%3E
>> > >
>> >
>>
>

Re: [VOTE] Moving Submarine to a separate Apache project proposal

[Wangda Tan]

I think I missed Rohith:

* Rohith Sharma K S (rohithsharm...@apache.org) (Hadoop PMC)

To add to initial committers.

On Fri, Sep 6, 2019 at 10:56 PM Dinesh Chitlangia 
wrote:

> +1
>
> -Dinesh
>
>
>
>
> On Fri, Sep 6, 2019 at 11:23 PM 俊平堵  wrote:
>
>> +1. Please include me also.
>>
>> Thanks,
>>
>> Junping
>>
>> Wangda Tan  于2019年9月1日周日 下午1:19写道：
>>
>> > Hi all,
>> >
>> > As we discussed in the previous thread [1],
>> >
>> > I just moved the spin-off proposal to CWIKI and completed all TODO
>> parts.
>> >
>> >
>> >
>> https://cwiki.apache.org/confluence/display/HADOOP/Submarine+Project+Spin-Off+to+TLP+Proposal
>> >
>> > If you have interests to learn more about this. Please review the
>> proposal
>> > let me know if you have any questions/suggestions for the proposal. This
>> > will be sent to board post voting passed. (And please note that the
>> > previous voting thread [2] to move Submarine to a separate Github repo
>> is a
>> > necessary effort to move Submarine to a separate Apache project but not
>> > sufficient so I sent two separate voting thread.)
>> >
>> > Please let me know if I missed anyone in the proposal, and reply if
>> you'd
>> > like to be included in the project.
>> >
>> > This voting runs for 7 days and will be concluded at Sep 7th, 11 PM PDT.
>> >
>> > Thanks,
>> > Wangda Tan
>> >
>> > [1]
>> >
>> >
>> https://lists.apache.org/thread.html/4a2210d567cbc05af92c12aa6283fd09b857ce209d537986ed800029@%3Cyarn-dev.hadoop.apache.org%3E
>> > [2]
>> >
>> >
>> https://lists.apache.org/thread.html/6e94469ca105d5a15dc63903a541bd21c7ef70b8bcff475a16b5ed73@%3Cyarn-dev.hadoop.apache.org%3E
>> >
>>
>

Re: [VOTE] Moving Submarine to a separate Apache project proposal

[Wangda Tan]

Thanks everyone for voting! And for whoever has interests to join
Submarine, you're always welcome!

And thanks to Owen for the kind help offered, I just added you to PMC list
in the proposal. It will be a great help to the community if you could
join!

For existing Hadoop committers who have interests to join, I plan to add
you to the initial list after discussed with other proposed initial
Submarine PMC members. The list I saw is:

* Naganarasimha Garla (naganarasimha_gr at apache dot org) (Hadoop PMC)
* Devaraj K (devaraj at apache dot org) (Hadoop PMC)
* Rakesh Radhakrishnan (rakeshr at apache dot org) (bookkeeper PMC, Hadoop
PMC, incubator, Mnemonic PMC, Zookeeper PMC)
* Vinayakumar B (vinayakumarb at apache dot org) (Hadoop PMC, incubator PMC)
* Ayush Saxena (ayushsaxena at apache dot org) (Hadoop Committer)
* Bibin Chundatt (bibinchundatt at apache dot org) (Hadoop PMC)
* Bharat Viswanadham (bharat at apache dot org) (Hadoop)
* Brahma Reddy Battula (brahma at apache dot org)) (Hadoop PMC)
* Abhishek Modi (abmodi at apache dot org) (Hadoop Committer)
* Wei-Chiu Chuang (weichiu at apache dot org) (Hadoop PMC)
* Junping Du (junping_du at apache dot org) (Hadoop PMC, member)

We'd like to see some reasonable contributions to the projects from all our
committers who will join now. Please join the weekly call or mailing lists
(once established) and share your inputs to the project. Members of
Submarine will reach out to all of you individually to understand the areas
you wish to contribute and will help in same. please let me know if you
DON'T want to add to the committer list.

Best,
Wangda Tan

On Fri, Sep 6, 2019 at 3:54 PM Wei-Chiu Chuang  wrote:

> +1
> I've involved myself in Submarine dev and I'd like to be included in the
> future.
>
> Thanks
>
> On Sat, Sep 7, 2019 at 5:27 AM Owen O'Malley 
> wrote:
>
>> Since you don't have any Apache Members, I'll join to provide Apache
>> oversight.
>>
>> .. Owen
>>
>> On Fri, Sep 6, 2019 at 1:38 PM Owen O'Malley 
>> wrote:
>>
>> > +1 for moving to a new project.
>> >
>> > On Sat, Aug 31, 2019 at 10:19 PM Wangda Tan 
>> wrote:
>> >
>> >> Hi all,
>> >>
>> >> As we discussed in the previous thread [1],
>> >>
>> >> I just moved the spin-off proposal to CWIKI and completed all TODO
>> parts.
>> >>
>> >>
>> >>
>> https://cwiki.apache.org/confluence/display/HADOOP/Submarine+Project+Spin-Off+to+TLP+Proposal
>> >>
>> >> If you have interests to learn more about this. Please review the
>> proposal
>> >> let me know if you have any questions/suggestions for the proposal.
>> This
>> >> will be sent to board post voting passed. (And please note that the
>> >> previous voting thread [2] to move Submarine to a separate Github repo
>> is
>> >> a
>> >> necessary effort to move Submarine to a separate Apache project but not
>> >> sufficient so I sent two separate voting thread.)
>> >>
>> >> Please let me know if I missed anyone in the proposal, and reply if
>> you'd
>> >> like to be included in the project.
>> >>
>> >> This voting runs for 7 days and will be concluded at Sep 7th, 11 PM
>> PDT.
>> >>
>> >> Thanks,
>> >> Wangda Tan
>> >>
>> >> [1]
>> >>
>> >>
>> https://lists.apache.org/thread.html/4a2210d567cbc05af92c12aa6283fd09b857ce209d537986ed800029@%3Cyarn-dev.hadoop.apache.org%3E
>> >> [2]
>> >>
>> >>
>> https://lists.apache.org/thread.html/6e94469ca105d5a15dc63903a541bd21c7ef70b8bcff475a16b5ed73@%3Cyarn-dev.hadoop.apache.org%3E
>> >>
>> >
>>
>

Re: [VOTE] Moving Submarine to a separate Apache project proposal

[Wangda Tan]

Hi Sree,

I put it to the proposal, please let me know what you think:

The traditional path at Apache would have been to create an incubator
> project, but the code is already being released by Apache and most of the
> developers are familiar with Apache rules and guidelines. In particular,
> the proposed PMC has 2 Apache TLP PMCs and proposed initial committers
> have 4 Apache TLP PMCs (from 3 different companies). They will provide
> oversight and guidance for the developers that are less experienced in the
> Apache Way. Therefore, the Submarine project would like to propose becoming
> a Top Level Project at Apache.
>

To me, putting to TLP has mostly pros, it is an easier process (same as ORC
spin-off from Hive), much less overhead to both dev community and Apache
side.

Thanks,
Wangda

On Sun, Sep 1, 2019 at 2:04 PM Sree Vaddi  wrote:

> +1 to move submarine to separate apache project.
>
> It is not clear in the proposal, if submarine majority voted to move to a
> seperate apache project,
> will it go through the incubation and TLP (top level project) later ?
> 1. Incubation
> pros and cons
> efforts towards making it a TLP
>
> 2. direct TLP
>
>
> Thank you.
> /Sree
>
>
>
>
> On Saturday, August 31, 2019, 10:19:06 PM PDT, Wangda Tan <
> wheele...@gmail.com> wrote:
>
>
> Hi all,
>
> As we discussed in the previous thread [1],
>
> I just moved the spin-off proposal to CWIKI and completed all TODO parts.
>
>
> https://cwiki.apache.org/confluence/display/HADOOP/Submarine+Project+Spin-Off+to+TLP+Proposal
>
> If you have interests to learn more about this. Please review the proposal
> let me know if you have any questions/suggestions for the proposal. This
> will be sent to board post voting passed. (And please note that the
> previous voting thread [2] to move Submarine to a separate Github repo is a
> necessary effort to move Submarine to a separate Apache project but not
> sufficient so I sent two separate voting thread.)
>
> Please let me know if I missed anyone in the proposal, and reply if you'd
> like to be included in the project.
>
> This voting runs for 7 days and will be concluded at Sep 7th, 11 PM PDT.
>
> Thanks,
> Wangda Tan
>
> [1]
>
> https://lists.apache.org/thread.html/4a2210d567cbc05af92c12aa6283fd09b857ce209d537986ed800029@%3Cyarn-dev.hadoop.apache.org%3E
> [2]
>
> https://lists.apache.org/thread.html/6e94469ca105d5a15dc63903a541bd21c7ef70b8bcff475a16b5ed73@%3Cyarn-dev.hadoop.apache.org%3E
>

[VOTE] Moving Submarine to a separate Apache project proposal

[Wangda Tan]

Hi all,

As we discussed in the previous thread [1],

I just moved the spin-off proposal to CWIKI and completed all TODO parts.

https://cwiki.apache.org/confluence/display/HADOOP/Submarine+Project+Spin-Off+to+TLP+Proposal

If you have interests to learn more about this. Please review the proposal
let me know if you have any questions/suggestions for the proposal. This
will be sent to board post voting passed. (And please note that the
previous voting thread [2] to move Submarine to a separate Github repo is a
necessary effort to move Submarine to a separate Apache project but not
sufficient so I sent two separate voting thread.)

Please let me know if I missed anyone in the proposal, and reply if you'd
like to be included in the project.

This voting runs for 7 days and will be concluded at Sep 7th, 11 PM PDT.

Thanks,
Wangda Tan

[1]
https://lists.apache.org/thread.html/4a2210d567cbc05af92c12aa6283fd09b857ce209d537986ed800029@%3Cyarn-dev.hadoop.apache.org%3E
[2]
https://lists.apache.org/thread.html/6e94469ca105d5a15dc63903a541bd21c7ef70b8bcff475a16b5ed73@%3Cyarn-dev.hadoop.apache.org%3E

Re: [VOTE] Move Submarine source code, documentation, etc. to a separate Apache Git repo

[Wangda Tan]

Got some trouble in creating git repo using Apache self-service. Created
https://issues.apache.org/jira/browse/INFRA-18964

Thanks,
Wangda

On Thu, Aug 29, 2019 at 11:52 PM Wangda Tan  wrote:

> Thanks everyone for voting.
>
> We got 12 binding +1s from committers. The vote passed.
>
> We will go ahead to create a repo "submarine", and will allow existing
> Hadoop committers to commit code to submarine.
>
> Best,
> Wangda
>
> On Thu, Aug 29, 2019 at 10:50 PM Arpit Agarwal 
> wrote:
>
>> +1
>>
>> > On Aug 23, 2019, at 7:05 PM, Wangda Tan  wrote:
>> >
>> > Hi devs,
>> >
>> > This is a voting thread to move Submarine source code, documentation
>> from
>> > Hadoop repo to a separate Apache Git repo. Which is based on
>> discussions of
>> >
>> https://lists.apache.org/thread.html/e49d60b2e0e021206e22bb2d430f4310019a8b29ee5020f3eea3bd95@%3Cyarn-dev.hadoop.apache.org%3E
>> >
>> > Contributors who have permissions to push to Hadoop Git repository will
>> > have permissions to push to the new Submarine repository.
>> >
>> > This voting thread will run for 7 days and will end at Aug 30th.
>> >
>> > Please let me know if you have any questions.
>> >
>> > Thanks,
>> > Wangda Tan
>>
>>

Re: [VOTE] Move Submarine source code, documentation, etc. to a separate Apache Git repo

[Wangda Tan]

Thanks everyone for voting.

We got 12 binding +1s from committers. The vote passed.

We will go ahead to create a repo "submarine", and will allow existing
Hadoop committers to commit code to submarine.

Best,
Wangda

On Thu, Aug 29, 2019 at 10:50 PM Arpit Agarwal 
wrote:

> +1
>
> > On Aug 23, 2019, at 7:05 PM, Wangda Tan  wrote:
> >
> > Hi devs,
> >
> > This is a voting thread to move Submarine source code, documentation from
> > Hadoop repo to a separate Apache Git repo. Which is based on discussions
> of
> >
> https://lists.apache.org/thread.html/e49d60b2e0e021206e22bb2d430f4310019a8b29ee5020f3eea3bd95@%3Cyarn-dev.hadoop.apache.org%3E
> >
> > Contributors who have permissions to push to Hadoop Git repository will
> > have permissions to push to the new Submarine repository.
> >
> > This voting thread will run for 7 days and will end at Aug 30th.
> >
> > Please let me know if you have any questions.
> >
> > Thanks,
> > Wangda Tan
>
>

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Wangda Tan]

I just put it to CWIKI:
https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches

Any thoughts? If no I will go ahead to send it to user/dev/general email
lists. (Should we send it to general?)

On Wed, Aug 28, 2019 at 5:03 PM Wangda Tan  wrote:

> Hi folks,
> Thanks all for sharing thoughts.
>
> I suggest moving branch-2 last release related discussion to a separate
> thread.
>
> We got 10 binding +1s. And no -1 to the proposal. I'm going to put this to
> WIKI first. And feel free to create a ticket and update our website.
>
> Thanks,
> Wangda
>
> On Wed, Aug 28, 2019 at 2:45 AM Konstantin Shvachko 
> wrote:
>
>> +1 for the proposal.
>>
>> I thought we already EOL-ed 2.6 though some time ago.
>>
>> Thanks,
>> --Konstantin
>>
>> On Tue, Aug 20, 2019 at 8:03 PM Wangda Tan  wrote:
>>
>> > Hi all,
>> >
>> > This is a vote thread to mark any versions smaller than 2.7 (inclusive),
>> > and 3.0 EOL. This is based on discussions of [1]
>> >
>> > This discussion runs for 7 days and will conclude on Aug 28 Wed.
>> >
>> > Please feel free to share your thoughts.
>> >
>> > Thanks,
>> > Wangda
>> >
>> > [1]
>> >
>> >
>> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAD++eC=ou-tit1faob-dbecqe6ht7ede7t1dyra2p1yinpe...@mail.gmail.com%3e
>> > ,
>> >
>>
>

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Wangda Tan]

Hi folks,
Thanks all for sharing thoughts.

I suggest moving branch-2 last release related discussion to a separate
thread.

We got 10 binding +1s. And no -1 to the proposal. I'm going to put this to
WIKI first. And feel free to create a ticket and update our website.

Thanks,
Wangda

On Wed, Aug 28, 2019 at 2:45 AM Konstantin Shvachko 
wrote:

> +1 for the proposal.
>
> I thought we already EOL-ed 2.6 though some time ago.
>
> Thanks,
> --Konstantin
>
> On Tue, Aug 20, 2019 at 8:03 PM Wangda Tan  wrote:
>
> > Hi all,
> >
> > This is a vote thread to mark any versions smaller than 2.7 (inclusive),
> > and 3.0 EOL. This is based on discussions of [1]
> >
> > This discussion runs for 7 days and will conclude on Aug 28 Wed.
> >
> > Please feel free to share your thoughts.
> >
> > Thanks,
> > Wangda
> >
> > [1]
> >
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAD++eC=ou-tit1faob-dbecqe6ht7ede7t1dyra2p1yinpe...@mail.gmail.com%3e
> > ,
> >
>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan]

Hi all,

We received comments and suggestions from contributors, committers and PMC
members regarding the proposal:
https://docs.google.com/document/d/1kE_f-r-ANh9qOeapdPwQPHhaJTS7IMiqDQAS8ESi4TA/edit?ts=5d529ec0

@Vinod Kumar Vavilapalli  could you provide suggestions
regarding what we should do next? Could you help to send this to the ASF
board?

Thanks,
Wangda Tan

On Tue, Aug 13, 2019 at 4:36 PM Wangda Tan  wrote:

> Hi folks,
>
> I just drafted a proposal which is targetted to send to PMC list and board
> for thoughts. Thanks Xun Liu for providing thoughts about future
> directions/architecture, and reviews from Keqiu Hu.
>
> Title: "Apache Submarine for Apache Top-Level Project"
>
>
> https://docs.google.com/document/d/1kE_f-r-ANh9qOeapdPwQPHhaJTS7IMiqDQAS8ESi4TA/edit
>
> I plan to send it to PMC list/board next Monday, so any
> comments/suggestions are welcome.
>
> Thanks,
> Wangda
>
>
> On Tue, Jul 30, 2019 at 6:01 PM 俊平堵  wrote:
>
>> Thanks Vinod for these great suggestions. I agree most of your comments
>> above.
>>  "For the Apache Hadoop community, this will be treated simply as
>> code-change and so need a committer +1?". IIUC, this should be treated as
>> feature branch merge, so may be 3 committer +1 is needed here according to
>> https://hadoop.apache.org/bylaws.html?
>>
>> bq. Can somebody who have cycles and been on the ASF lists for a while
>> look into the process here?
>> I can check with ASF members who has experience on this if no one haven't
>> yet.
>>
>> Thanks,
>>
>> Junping
>>
>> Vinod Kumar Vavilapalli  于2019年7月29日周一 下午9:46写道：
>>
>>> Looks like there's a meaningful push behind this.
>>>
>>> Given the desire is to fork off Apache Hadoop, you'd want to make sure
>>> this enthusiasm turns into building a real, independent but more
>>> importantly a sustainable community.
>>>
>>> Given that there were two official releases off the Apache Hadoop
>>> project, I doubt if you'd need to go through the incubator process. Instead
>>> you can directly propose a new TLP at ASF board. The last few times this
>>> happened was with ORC, and long before that with Hive, HBase etc. Can
>>> somebody who have cycles and been on the ASF lists for a while look into
>>> the process here?
>>>
>>> For the Apache Hadoop community, this will be treated simply as
>>> code-change and so need a committer +1? You can be more gently by formally
>>> doing a vote once a process doc is written down.
>>>
>>> Back to the sustainable community point, as part of drafting this
>>> proposal, you'd definitely want to make sure all of the Apache Hadoop
>>> PMC/Committers can exercise their will to join this new project as
>>> PMC/Committers respectively without any additional constraints.
>>>
>>> Thanks
>>> +Vinod
>>>
>>> > On Jul 25, 2019, at 1:31 PM, Wangda Tan  wrote:
>>> >
>>> > Thanks everybody for sharing your thoughts. I saw positive feedbacks
>>> from
>>> > 20+ contributors!
>>> >
>>> > So I think we should move it forward, any suggestions about what we
>>> should
>>> > do?
>>> >
>>> > Best,
>>> > Wangda
>>> >
>>> > On Mon, Jul 22, 2019 at 5:36 PM neo  wrote:
>>> >
>>> >> +1, This is neo from TiDB & TiKV community.
>>> >> Thanks Xun for bring this up.
>>> >>
>>> >> Our CNCF project's open source distributed KV storage system TiKV,
>>> >> Hadoop submarine's machine learning engine helps us to optimize data
>>> >> storage,
>>> >> helping us solve some problems in data hotspots and data shuffers.
>>> >>
>>> >> We are ready to improve the performance of TiDB in our open source
>>> >> distributed relational database TiDB and also using the hadoop
>>> submarine
>>> >> machine learning engine.
>>> >>
>>> >> I think if submarine can be independent, it will develop faster and
>>> better.
>>> >> Thanks to the hadoop community for developing submarine!
>>> >>
>>> >> Best Regards,
>>> >> neo
>>> >> www.pingcap.com / https://github.com/pingcap/tidb /
>>> >> https://github.com/tikv
>>> >>
>>> >> Xun Liu  于2019年7月22日周一 下午4:07写道：
>>> >>
>>> >>> @adam.antal
>>> >>>
>&g

[VOTE] Move Submarine source code, documentation, etc. to a separate Apache Git repo

[Wangda Tan]

Hi devs,

This is a voting thread to move Submarine source code, documentation from
Hadoop repo to a separate Apache Git repo. Which is based on discussions of
https://lists.apache.org/thread.html/e49d60b2e0e021206e22bb2d430f4310019a8b29ee5020f3eea3bd95@%3Cyarn-dev.hadoop.apache.org%3E

Contributors who have permissions to push to Hadoop Git repository will
have permissions to push to the new Submarine repository.

This voting thread will run for 7 days and will end at Aug 30th.

Please let me know if you have any questions.

Thanks,
Wangda Tan

Re: Hadoop Community Sync Up Schedule

[Wangda Tan]

Hi all,

I just updated schedules, which takes effect next week. now it looks like:

[image: image.png]

I made sure that no two YARN sync ups nor two HDFS sync ups on the same day.

Please check the calendar to see if you have any questions:
https://calendar.google.com/calendar/b/3?cid=aGFkb29wLmNvbW11bml0eS5zeW5jLnVwQGdtYWlsLmNvbQ
<https://calendar.google.com/calendar/b/3?cid=aGFkb29wLmNvbW11bml0eS5zeW5jLnVwQGdtYWlsLmNvbQ>

Thanks,
Wangda



On Fri, Aug 23, 2019 at 1:50 AM Matt Foley  wrote:

> Wangda and Eric,
> We can express the intent, I think, by scheduling two recurring meetings:
> - monthly, on the 2nd Wednesday, and
> - monthly, on the 4th Wednesday.
>
> This is pretty easy to understand, and not too onerous to maintain.
> But I’m okay with simple bi-weekly too.
>
> I’m neutral on 10 vs 11am, PDT.  But do all participants presently use
> Daylight Savings and change on the same date?  Because I can’t conveniently
> do 9am PST, so if it needs to stay fixed in, say, India Timezone, then 11am
> PDT / 10am PST would be best.
> —Matt
>
> On Aug 21, 2019, at 7:33 AM, epa...@apache.org wrote:
>
> Let's go with bi-weekly (every 2 weeks). Sometimes this gives us 3
> sync-ups in one month, which I think is fine.
> -Eric Payne
>
> On Wednesday, August 21, 2019, 5:01:52 AM CDT, Wangda Tan <
> wheele...@gmail.com <mailto:wheele...@gmail.com>> wrote:
> >
> > For folks in other US time zones: how about 11am PDT, is it better or
> 10am
> > PDT will be better? I will be fine with both.
> >
> > Hi Matt,
> >
> > Thanks for mentioning this issue, this is the exactly issue I saw 藍.
> >
> > Basically there’re two options:
> >
> > - a. weekly, bi-weekly (for odd/even week) and every four months.
> > - b. weekly, 1st/3rd week or 2nd/4th week, x-th week monthly.
> >
> > I’m not sure which one is easier for people to understand as the issue
> you
> > mentioned.
> >
> > After thinking about it. I prefer a. since it is more consistent for
> > audience and not disrupted because of calendar.
> >
> > If we choose a. I will redo the proposal and make it aligns with a.
> >
> > Thoughts?
> >
> > Thanks,
> > Wangda
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org  common-dev-unsubscr...@hadoop.apache.org>
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> <mailto:common-dev-h...@hadoop.apache.org>
>

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Wangda Tan]

Hi Steve,

The proposal is to EOL for the following branches:

[2.0.x - 2.7.x]
[3.0.x]

2.8.x, 2.9.x, 2.10.x (not released yet), 3.1.x (and later) are not EOL.

Thoughts?

On Sat, Aug 24, 2019 at 1:40 AM Steve Loughran  wrote:

>
>
> On Wed, Aug 21, 2019 at 4:03 AM Wangda Tan  wrote:
>
>> Hi all,
>>
>> This is a vote thread to mark any versions smaller than 2.7 (inclusive),
>> and 3.0 EOL. This is based on discussions of [1]
>>
>
> 3.0 inclusive? i.e the non EOl ones being 2.8+ and 3.1+?
>

Re: Hadoop Community Sync Up Schedule

[Wangda Tan]

Sounds good, let me make the changes to do simply bi-weekly then.

I will update it tonight if possible.

Best,
Wangda

On Fri, Aug 23, 2019 at 1:50 AM Matt Foley  wrote:

> Wangda and Eric,
> We can express the intent, I think, by scheduling two recurring meetings:
> - monthly, on the 2nd Wednesday, and
> - monthly, on the 4th Wednesday.
>
> This is pretty easy to understand, and not too onerous to maintain.
> But I’m okay with simple bi-weekly too.
>
> I’m neutral on 10 vs 11am, PDT.  But do all participants presently use
> Daylight Savings and change on the same date?  Because I can’t conveniently
> do 9am PST, so if it needs to stay fixed in, say, India Timezone, then 11am
> PDT / 10am PST would be best.
> —Matt
>
> On Aug 21, 2019, at 7:33 AM, epa...@apache.org wrote:
>
> Let's go with bi-weekly (every 2 weeks). Sometimes this gives us 3
> sync-ups in one month, which I think is fine.
> -Eric Payne
>
> On Wednesday, August 21, 2019, 5:01:52 AM CDT, Wangda Tan <
> wheele...@gmail.com <mailto:wheele...@gmail.com>> wrote:
> >
> > For folks in other US time zones: how about 11am PDT, is it better or
> 10am
> > PDT will be better? I will be fine with both.
> >
> > Hi Matt,
> >
> > Thanks for mentioning this issue, this is the exactly issue I saw 藍.
> >
> > Basically there’re two options:
> >
> > - a. weekly, bi-weekly (for odd/even week) and every four months.
> > - b. weekly, 1st/3rd week or 2nd/4th week, x-th week monthly.
> >
> > I’m not sure which one is easier for people to understand as the issue
> you
> > mentioned.
> >
> > After thinking about it. I prefer a. since it is more consistent for
> > audience and not disrupted because of calendar.
> >
> > If we choose a. I will redo the proposal and make it aligns with a.
> >
> > Thoughts?
> >
> > Thanks,
> > Wangda
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org  common-dev-unsubscr...@hadoop.apache.org>
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> <mailto:common-dev-h...@hadoop.apache.org>
>

Re: Thoughts about moving submarine to a separate git repo?

[Wangda Tan]

>
> One major point from my end is that the commits to new repo ideally should
> happen from committers or branch committers.


Definitely agree!

On Thu, Aug 22, 2019 at 12:32 PM Sunil Govindan  wrote:

> Thanks Wangda for sharing the thoughts.
>
> I agree with the idea of new repo for more cleaner code base and remove
> additional dependencies, jenkins etc.
> One major point from my end is that the commits to new repo ideally should
> happen from committers or branch committers.
>
> As hadoop community, we could help in this case. @Wangda Tan
>   your thoughts?
>
> Thanks
> Sunil
>
> On Wed, Aug 21, 2019 at 9:47 AM Wangda Tan  wrote:
>
>> Hi Xun,
>>
>> Thanks for starting this thread. I'm glad to see the existing momentum
>> made
>> by Submarine community, and I like the proposal to make it be a separate
>> Git repo.
>>
>> Some suggestions:
>>
>> 1) For options mentioned by Sunil: I think it's better to be a separate
>> Git
>> repo instead of a branch. To me, the branch is targeted for a
>> diverged codebase instead of a new codebase. Since Submarine needs a clean
>> root source code directory. I think moving to a new Git repo makes more
>> sense.
>>
>> 2) For the Submarine external code, when we pulling them in, I think we
>> need to make sure license, iCLA, code comply with Apache standard. Which
>> means we need to do some additional reviews, etc. for patches being pulled
>> in. (instead of as-is).
>>
>> 3) Can we address comments from @Wei-Chiu Chuang  ,
>> to
>> give some extra time for existing Hadoop committers/contributors who have
>> interests to review the code? Waiting for at least 1 day for a big patch
>> and 6 hours for a minor fix might be a good rule to follow. And @Wei-Chiu
>> Chuang  please let Submarine community know if you
>> have
>> anything interested to review so developers can ping you when they have
>> any
>> patches.
>>
>> Best,
>> Wangda
>>
>> On Tue, Aug 20, 2019 at 9:54 PM Wei-Chiu Chuang 
>> wrote:
>>
>> > >
>> > >
>> > >
>> > > Submarine dev community has a total of 8 developers and submits an
>> > average
>> > > of 4 to 5 PR per day.
>> > > But there are a limited number of Hadoop committer actively help
>> review
>> > and
>> > > merge patches, which causes development progress delays.
>> > >
>> > > I just want to point this out that this is concerning -- I wanted to
>> help
>> > review patches, but it wasn't obvious the patches were raised as PRs in
>> a
>> > non-apache git repo.
>> > Please be more inclusive.
>> >
>>
>

Re: Hadoop Community Sync Up Schedule

[Wangda Tan]

For folks in other US time zones: how about 11am PDT, is it better or 10am
PDT will be better? I will be fine with both.

Hi Matt,

Thanks for mentioning this issue, this is the exactly issue I saw 藍.

Basically there’re two options:

- a. weekly, bi-weekly (for odd/even week) and every four months.
- b. weekly, 1st/3rd week or 2nd/4th week, x-th week monthly.

I’m not sure which one is easier for people to understand as the issue you
mentioned.

After thinking about it. I prefer a. since it is more consistent for
audience and not disrupted because of calendar.

If we choose a. I will redo the proposal and make it aligns with a.

Thoughts?

Thanks,
Wangda



On Wed, Aug 21, 2019 at 4:11 AM Matt Foley  wrote:

> Hi Wangda, thanks for this. A question about the schedule correction:
>
> > > 1) In the proposal, repeats are not properly. (I used bi-weekly
> instead of
> > 2nd/4th week as repeat frequency). I'd like to fix the frequency on Thu
> and
> > it will take effect starting next week.
>
> I understand that “bi-weekly” may cause 3 meetings in some months, and
> that’s not the intent.
> Is it your intent to schedule “Wednesday of the 2nd and 4th weeks of each
> month” or “2nd and 4th Wednesday of each month”? — which of course are not
> the same, for months that start on Thu-Sat...
>
> As a side note, I find that my calendar program cannot express “Wednesday
> of the second week of the month”, but does know how to do “second Wednesday
> of the month”.
>
> But I’ll schedule them one-by-one if I have to, I just wanted clarity. :-)
>
> Thanks,
> —Matt
>
>
> On Aug 19, 2019, at 8:31 PM, Wangda Tan  wrote:
>
> Hi folks,
>
> We have run community sync up for 1.5 months. I spoke to folks offline and
> got some feedback. Here's a summary of what I've observed from sync ups and
> talked to organizers.
>
> Following sync ups have very good participants (sometimes 10+ folks
> joined):
> - YARN/MR monthly sync up in APAC (Mandarin)
> - HDFS monthly sync up in APAC (Mandarin).
> - Submarine weekly sync up in APAC (Mandarin).
>
> Following sync up have OK-ish participants: (3-5 folks joined).
> - Storage monthly sync up in APAC (English)
> - Storage bi-weekly sync up in US (English)
> - YARN bi-weekly sync up in US (English).
>
> Following sync ups don't have good participants: (Skipped a couple of
> times).
> - YARN monthly sync up in APAC (English).
> - Submarine bi-weekly sync up in US (English).
>
> *So I'd like to propose the following changes and fixes of the schedule: *
> 1) Cancel the YARN/MR monthly sync up in APAC (English). Folks from APAC
> who speak English can choose to join the US session.
> 2) Cancel the Submarine bi-weekly sync up in US (English). Now Submarine
> developers and users are fast-growing in Mandarin-speaking areas. We can
> resume the sync if we do see demands from English-speaking areas.
> 3) Update the US sync up time from 9AM to 10AM PDT. 9AM is too early for
> most of the west-cost folks.
>
> *Following are fixes for the schedule:  *
> 1) In the proposal, repeats are not properly. (I used bi-weekly instead of
> 2nd/4th week as repeat frequency). I'd like to fix the frequency on Thu and
> it will take effect starting next week.
>
> Overall, thanks for everybody who participated in the sync ups. I do see
> community contributions grow in the last one month!
>
> Any thoughts about the proposal?
>
> Thanks,
> Wangda
>
>
>
>
> On Thu, Jul 25, 2019 at 11:53 AM 俊平堵  wrote:
>
> > Hi Folks,
> >
> > Kindly remind that we have YARN+MR APAC sync today, and you are
> > welcome to join:
> >
> >
> > Time and Date：07/25 1:00 pm (CST Time)
> >
> > Zoom link：Zoom | https://cloudera.zoom.us/j/880548968
> >
> > Summary：
> >
> >
> https://docs.google.com/document/d/1GY55sXrekVd-aDyRY7uzaX0hMDPyh3T-AL1kUY2TI5M
> >
> >
> > Thanks,
> >
> >
> > Junping
> >
> >
> >
> > Wangda Tan  于2019年6月28日周五 上午2:57写道：
> >
> >> Hi folks,
> >>
> >> Here's the Hadoop Community Sync Up proposal/schedule:
> >>
> >
> https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#heading=h.xh4zfwj8ppmn
> >>
> >> And here's calendar file:
> >>
> >>
> >>
> >
> https://calendar.google.com/calendar/ical/hadoop.community.sync.up%40gmail.com/public/basic.ics
> >>
> >> We gave it a try this week for YARN+MR and Submarine sync, feedbacks
> from
> >> participants seems pretty good, lots of new information shared during
> > sync
> >> up, and companies are using/developing Hadoop can better know each
> other.
> >>
> >> Next week there're 4 community sync-ups (Two Submarine for different
> >> timezones, one YARN+MR, one storage), please join to whichever you're
> >> interested:
> >>
> >> [image: image.png]
> >>
> >> Zoom info and notes can be found in the Google calendar invitation.
> >>
> >> Thanks,
> >> Wangda
> >>
> >
>
> --
Sent from Gmail Mobile

Re: Thoughts about moving submarine to a separate git repo?

[Wangda Tan]

Hi Xun,

Thanks for starting this thread. I'm glad to see the existing momentum made
by Submarine community, and I like the proposal to make it be a separate
Git repo.

Some suggestions:

1) For options mentioned by Sunil: I think it's better to be a separate Git
repo instead of a branch. To me, the branch is targeted for a
diverged codebase instead of a new codebase. Since Submarine needs a clean
root source code directory. I think moving to a new Git repo makes more
sense.

2) For the Submarine external code, when we pulling them in, I think we
need to make sure license, iCLA, code comply with Apache standard. Which
means we need to do some additional reviews, etc. for patches being pulled
in. (instead of as-is).

3) Can we address comments from @Wei-Chiu Chuang  , to
give some extra time for existing Hadoop committers/contributors who have
interests to review the code? Waiting for at least 1 day for a big patch
and 6 hours for a minor fix might be a good rule to follow. And @Wei-Chiu
Chuang  please let Submarine community know if you have
anything interested to review so developers can ping you when they have any
patches.

Best,
Wangda

On Tue, Aug 20, 2019 at 9:54 PM Wei-Chiu Chuang  wrote:

> >
> >
> >
> > Submarine dev community has a total of 8 developers and submits an
> average
> > of 4 to 5 PR per day.
> > But there are a limited number of Hadoop committer actively help review
> and
> > merge patches, which causes development progress delays.
> >
> > I just want to point this out that this is concerning -- I wanted to help
> review patches, but it wasn't obvious the patches were raised as PRs in a
> non-apache git repo.
> Please be more inclusive.
>

[VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Wangda Tan]

Hi all,

This is a vote thread to mark any versions smaller than 2.7 (inclusive),
and 3.0 EOL. This is based on discussions of [1]

This discussion runs for 7 days and will conclude on Aug 28 Wed.

Please feel free to share your thoughts.

Thanks,
Wangda

[1]
http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAD++eC=ou-tit1faob-dbecqe6ht7ede7t1dyra2p1yinpe...@mail.gmail.com%3e
,

Re: Make EOL branches to public?

[Wangda Tan]

Thank you all for suggestions. Let me send a vote email to mark 2.6, 2.7,
3.0 EOL.

- Wangda

On Wed, Aug 21, 2019 at 9:34 AM Akira Ajisaka  wrote:

> +1
>
> Thank you for the discussion.
>
> -Akira
>
> On Wed, Aug 21, 2019 at 5:51 AM Wei-Chiu Chuang 
> wrote:
> >
> > +1
> > I feel like one year of inactivity is a good sign that the community is
> not
> > interested in the branch any more.
> >
> > On Fri, Aug 16, 2019 at 3:14 AM Wangda Tan  wrote:
> >
> > > Hi folks,
> > >
> > > Want to hear your thoughts about what we should do to make some
> branches
> > > EOL. We discussed a couple of times before in dev lists and PMC list.
> > > However, we couldn't get a formal process of EOL. According to the
> > > discussion. It is hard to decide it based on time like "After 1st
> release,
> > > EOL in 2 years". Because community members still want to maintain it
> and
> > > developers still want to get a newer version released.
> > >
> > > However, without a public place to figure out which release will be
> EOL, it
> > > is very hard for users to choose the right releases to upgrade and
> develop.
> > >
> > > So I want to propose to make an agreement about making a public EOL
> wiki
> > > page and create a process to EOL a release:
> > >
> > > The process I'm thinking is very simple: If no volunteer to do a
> > > maintenance release in a short to mid-term (like 3 months to 1 or 1.5
> > > year). We will claim a release is EOL. After EOL community can still
> choose
> > > to do a security-only release.
> > >
> > > Here's a list which I can think about:
> > >
> > > 1) 2.6.x (Or any release older than 2.6) (Last released at Oct 2016)
> > > 2) 2.7.x (Last released at Apr 2018)
> > > 4) 3.0.x (Last released at May 2018)
> > >
> > > Thoughts?
> > >
> > > Thanks,
> > > Wangda
> > >
>

Re: Hadoop Community Sync Up Schedule

[Wangda Tan]

Hi folks,

We have run community sync up for 1.5 months. I spoke to folks offline and
got some feedback. Here's a summary of what I've observed from sync ups and
talked to organizers.

Following sync ups have very good participants (sometimes 10+ folks
joined):
- YARN/MR monthly sync up in APAC (Mandarin)
- HDFS monthly sync up in APAC (Mandarin).
- Submarine weekly sync up in APAC (Mandarin).

Following sync up have OK-ish participants: (3-5 folks joined).
- Storage monthly sync up in APAC (English)
- Storage bi-weekly sync up in US (English)
- YARN bi-weekly sync up in US (English).

Following sync ups don't have good participants: (Skipped a couple of
times).
- YARN monthly sync up in APAC (English).
- Submarine bi-weekly sync up in US (English).

*So I'd like to propose the following changes and fixes of the schedule: *
1) Cancel the YARN/MR monthly sync up in APAC (English). Folks from APAC
who speak English can choose to join the US session.
2) Cancel the Submarine bi-weekly sync up in US (English). Now Submarine
developers and users are fast-growing in Mandarin-speaking areas. We can
resume the sync if we do see demands from English-speaking areas.
3) Update the US sync up time from 9AM to 10AM PDT. 9AM is too early for
most of the west-cost folks.

*Following are fixes for the schedule:  *
1) In the proposal, repeats are not properly. (I used bi-weekly instead of
2nd/4th week as repeat frequency). I'd like to fix the frequency on Thu and
it will take effect starting next week.

Overall, thanks for everybody who participated in the sync ups. I do see
community contributions grow in the last one month!

Any thoughts about the proposal?

Thanks,
Wangda




On Thu, Jul 25, 2019 at 11:53 AM 俊平堵  wrote:

> Hi Folks,
>
>  Kindly remind that we have YARN+MR APAC sync today, and you are
> welcome to join:
>
>
> Time and Date：07/25 1:00 pm (CST Time)
>
> Zoom link：Zoom | https://cloudera.zoom.us/j/880548968
>
> Summary：
>
> https://docs.google.com/document/d/1GY55sXrekVd-aDyRY7uzaX0hMDPyh3T-AL1kUY2TI5M
>
>
> Thanks,
>
>
> Junping
>
>
>
> Wangda Tan  于2019年6月28日周五 上午2:57写道：
>
> > Hi folks,
> >
> > Here's the Hadoop Community Sync Up proposal/schedule:
> >
> https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#heading=h.xh4zfwj8ppmn
> >
> > And here's calendar file:
> >
> >
> >
> https://calendar.google.com/calendar/ical/hadoop.community.sync.up%40gmail.com/public/basic.ics
> >
> > We gave it a try this week for YARN+MR and Submarine sync, feedbacks from
> > participants seems pretty good, lots of new information shared during
> sync
> > up, and companies are using/developing Hadoop can better know each other.
> >
> > Next week there're 4 community sync-ups (Two Submarine for different
> > timezones, one YARN+MR, one storage), please join to whichever you're
> > interested:
> >
> > [image: image.png]
> >
> > Zoom info and notes can be found in the Google calendar invitation.
> >
> > Thanks,
> > Wangda
> >
>

Make EOL branches to public?

[Wangda Tan]

Hi folks,

Want to hear your thoughts about what we should do to make some branches
EOL. We discussed a couple of times before in dev lists and PMC list.
However, we couldn't get a formal process of EOL. According to the
discussion. It is hard to decide it based on time like "After 1st release,
EOL in 2 years". Because community members still want to maintain it and
developers still want to get a newer version released.

However, without a public place to figure out which release will be EOL, it
is very hard for users to choose the right releases to upgrade and develop.

So I want to propose to make an agreement about making a public EOL wiki
page and create a process to EOL a release:

The process I'm thinking is very simple: If no volunteer to do a
maintenance release in a short to mid-term (like 3 months to 1 or 1.5
year). We will claim a release is EOL. After EOL community can still choose
to do a security-only release.

Here's a list which I can think about:

1) 2.6.x (Or any release older than 2.6) (Last released at Oct 2016)
2) 2.7.x (Last released at Apr 2018)
4) 3.0.x (Last released at May 2018)

Thoughts?

Thanks,
Wangda

Re: [DISCUSS] Hadoop 2019 Release Planning

[Wangda Tan]

Thanks all for the suggestions and volunteering run these releases:

I just updated roadmap of Hadoop wiki:
https://cwiki.apache.org/confluence/display/HADOOP/Roadmap

Now I put 5 releases and planned date is 2019. We have 3 releases have
volunteered RM (Release Manager):

- 3.2.1: Rohith
- 2.8.6: Junping
- 2.10.0: Jonathan

We still need RM for 3.3.0 and 3.1.3.

Also, from my past experience, it will be helpful to get a co-RM (or shadow
RM) to do release together since release will have some effort which two
RMs can share some workload.

Can you help to update Roadmap wiki and put an estimated release date,
feature freeze date, etc. Which we can starting release cycle sooner if
possible?

Thanks,
Wangda


On Fri, Aug 16, 2019 at 5:00 PM Rohith Sharma K S 
wrote:

> Hi Wangda,
>
> Thanks for initiating this. I would like to nominate myself for 3.2.1
> Release Management.
>
> -Rohith Sharma K S
>
> On Sat, 10 Aug 2019 at 08:29, Wangda Tan  wrote:
>
> > Hi all,
> >
> > Hope this email finds you well
> >
> > I want to hear your thoughts about what should be the release plan for
> > 2019.
> >
> > In 2018, we released:
> > - 1 maintenance release of 2.6
> > - 3 maintenance releases of 2.7
> > - 3 maintenance releases of 2.8
> > - 3 releases of 2.9
> > - 4 releases of 3.0
> > - 2 releases of 3.1
> >
> > Total 16 releases in 2018.
> >
> > In 2019, by far we only have two releases:
> > - 1 maintenance release of 3.1
> > - 1 minor release of 3.2.
> >
> > However, the community put a lot of efforts to stabilize features of
> > various release branches.
> > There're:
> > - 217 fixed patches in 3.1.3 [1]
> > - 388 fixed patches in 3.2.1 [2]
> > - 1172 fixed patches in 3.3.0 [3] (OMG!)
> >
> > I think it is the time to do maintenance releases of 3.1/3.2 and do a
> minor
> > release for 3.3.0.
> >
> > In addition, I saw community discussion to do a 2.8.6 release for
> security
> > fixes.
> >
> > Any other releases? I think there're release plans for Ozone as well. And
> > please add your thoughts.
> >
> > Volunteers welcome! If you have interests to run a release as Release
> > Manager (or co-Resource Manager), please respond to this email thread so
> we
> > can coordinate.
> >
> > Thanks,
> > Wangda Tan
> >
> > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed AND
> > fixVersion = 3.1.3
> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed AND
> > fixVersion = 3.2.1
> > [3] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed AND
> > fixVersion = 3.3.0
> >
>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan]

Hi folks,

I just drafted a proposal which is targetted to send to PMC list and board
for thoughts. Thanks Xun Liu for providing thoughts about future
directions/architecture, and reviews from Keqiu Hu.

Title: "Apache Submarine for Apache Top-Level Project"

https://docs.google.com/document/d/1kE_f-r-ANh9qOeapdPwQPHhaJTS7IMiqDQAS8ESi4TA/edit

I plan to send it to PMC list/board next Monday, so any
comments/suggestions are welcome.

Thanks,
Wangda


On Tue, Jul 30, 2019 at 6:01 PM 俊平堵  wrote:

> Thanks Vinod for these great suggestions. I agree most of your comments
> above.
>  "For the Apache Hadoop community, this will be treated simply as
> code-change and so need a committer +1?". IIUC, this should be treated as
> feature branch merge, so may be 3 committer +1 is needed here according to
> https://hadoop.apache.org/bylaws.html?
>
> bq. Can somebody who have cycles and been on the ASF lists for a while
> look into the process here?
> I can check with ASF members who has experience on this if no one haven't
> yet.
>
> Thanks,
>
> Junping
>
> Vinod Kumar Vavilapalli  于2019年7月29日周一 下午9:46写道：
>
>> Looks like there's a meaningful push behind this.
>>
>> Given the desire is to fork off Apache Hadoop, you'd want to make sure
>> this enthusiasm turns into building a real, independent but more
>> importantly a sustainable community.
>>
>> Given that there were two official releases off the Apache Hadoop
>> project, I doubt if you'd need to go through the incubator process. Instead
>> you can directly propose a new TLP at ASF board. The last few times this
>> happened was with ORC, and long before that with Hive, HBase etc. Can
>> somebody who have cycles and been on the ASF lists for a while look into
>> the process here?
>>
>> For the Apache Hadoop community, this will be treated simply as
>> code-change and so need a committer +1? You can be more gently by formally
>> doing a vote once a process doc is written down.
>>
>> Back to the sustainable community point, as part of drafting this
>> proposal, you'd definitely want to make sure all of the Apache Hadoop
>> PMC/Committers can exercise their will to join this new project as
>> PMC/Committers respectively without any additional constraints.
>>
>> Thanks
>> +Vinod
>>
>> > On Jul 25, 2019, at 1:31 PM, Wangda Tan  wrote:
>> >
>> > Thanks everybody for sharing your thoughts. I saw positive feedbacks
>> from
>> > 20+ contributors!
>> >
>> > So I think we should move it forward, any suggestions about what we
>> should
>> > do?
>> >
>> > Best,
>> > Wangda
>> >
>> > On Mon, Jul 22, 2019 at 5:36 PM neo  wrote:
>> >
>> >> +1, This is neo from TiDB & TiKV community.
>> >> Thanks Xun for bring this up.
>> >>
>> >> Our CNCF project's open source distributed KV storage system TiKV,
>> >> Hadoop submarine's machine learning engine helps us to optimize data
>> >> storage,
>> >> helping us solve some problems in data hotspots and data shuffers.
>> >>
>> >> We are ready to improve the performance of TiDB in our open source
>> >> distributed relational database TiDB and also using the hadoop
>> submarine
>> >> machine learning engine.
>> >>
>> >> I think if submarine can be independent, it will develop faster and
>> better.
>> >> Thanks to the hadoop community for developing submarine!
>> >>
>> >> Best Regards,
>> >> neo
>> >> www.pingcap.com / https://github.com/pingcap/tidb /
>> >> https://github.com/tikv
>> >>
>> >> Xun Liu  于2019年7月22日周一 下午4:07写道：
>> >>
>> >>> @adam.antal
>> >>>
>> >>> The submarine development team has completed the following
>> preparations:
>> >>> 1. Established a temporary test repository on Github.
>> >>> 2. Change the package name of hadoop submarine from
>> org.hadoop.submarine
>> >> to
>> >>> org.submarine
>> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
>> >>> 4. On the Github docked travis-ci system, all test cases have been
>> >> tested;
>> >>> 5. Several Hadoop submarine users completed the system test using the
>> >> code
>> >>> in this repository.
>> >>>
>> >>> 赵欣  于2019年7月22日周一 上午9:38写道：
>> >>>
>> >>>> Hi
>> >>>>
>

[DISCUSS] Hadoop 2019 Release Planning

[Wangda Tan]

Hi all,

Hope this email finds you well

I want to hear your thoughts about what should be the release plan for
2019.

In 2018, we released:
- 1 maintenance release of 2.6
- 3 maintenance releases of 2.7
- 3 maintenance releases of 2.8
- 3 releases of 2.9
- 4 releases of 3.0
- 2 releases of 3.1

Total 16 releases in 2018.

In 2019, by far we only have two releases:
- 1 maintenance release of 3.1
- 1 minor release of 3.2.

However, the community put a lot of efforts to stabilize features of
various release branches.
There're:
- 217 fixed patches in 3.1.3 [1]
- 388 fixed patches in 3.2.1 [2]
- 1172 fixed patches in 3.3.0 [3] (OMG!)

I think it is the time to do maintenance releases of 3.1/3.2 and do a minor
release for 3.3.0.

In addition, I saw community discussion to do a 2.8.6 release for security
fixes.

Any other releases? I think there're release plans for Ozone as well. And
please add your thoughts.

Volunteers welcome! If you have interests to run a release as Release
Manager (or co-Resource Manager), please respond to this email thread so we
can coordinate.

Thanks,
Wangda Tan

[1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed AND
fixVersion = 3.1.3
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed AND
fixVersion = 3.2.1
[3] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed AND
fixVersion = 3.3.0

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan]

Thanks Vinod, the proposal to make it be TLP definitely a great suggestion.
I will draft a proposal and keep the thread posted.

Best,
Wangda

On Mon, Jul 29, 2019 at 3:46 PM Vinod Kumar Vavilapalli 
wrote:

> Looks like there's a meaningful push behind this.
>
> Given the desire is to fork off Apache Hadoop, you'd want to make sure
> this enthusiasm turns into building a real, independent but more
> importantly a sustainable community.
>
> Given that there were two official releases off the Apache Hadoop project,
> I doubt if you'd need to go through the incubator process. Instead you can
> directly propose a new TLP at ASF board. The last few times this happened
> was with ORC, and long before that with Hive, HBase etc. Can somebody who
> have cycles and been on the ASF lists for a while look into the process
> here?
>
> For the Apache Hadoop community, this will be treated simply as
> code-change and so need a committer +1? You can be more gently by formally
> doing a vote once a process doc is written down.
>
> Back to the sustainable community point, as part of drafting this
> proposal, you'd definitely want to make sure all of the Apache Hadoop
> PMC/Committers can exercise their will to join this new project as
> PMC/Committers respectively without any additional constraints.
>
> Thanks
> +Vinod
>
> > On Jul 25, 2019, at 1:31 PM, Wangda Tan  wrote:
> >
> > Thanks everybody for sharing your thoughts. I saw positive feedbacks from
> > 20+ contributors!
> >
> > So I think we should move it forward, any suggestions about what we
> should
> > do?
> >
> > Best,
> > Wangda
> >
> > On Mon, Jul 22, 2019 at 5:36 PM neo  wrote:
> >
> >> +1, This is neo from TiDB & TiKV community.
> >> Thanks Xun for bring this up.
> >>
> >> Our CNCF project's open source distributed KV storage system TiKV,
> >> Hadoop submarine's machine learning engine helps us to optimize data
> >> storage,
> >> helping us solve some problems in data hotspots and data shuffers.
> >>
> >> We are ready to improve the performance of TiDB in our open source
> >> distributed relational database TiDB and also using the hadoop submarine
> >> machine learning engine.
> >>
> >> I think if submarine can be independent, it will develop faster and
> better.
> >> Thanks to the hadoop community for developing submarine!
> >>
> >> Best Regards,
> >> neo
> >> www.pingcap.com / https://github.com/pingcap/tidb /
> >> https://github.com/tikv
> >>
> >> Xun Liu  于2019年7月22日周一 下午4:07写道：
> >>
> >>> @adam.antal
> >>>
> >>> The submarine development team has completed the following
> preparations:
> >>> 1. Established a temporary test repository on Github.
> >>> 2. Change the package name of hadoop submarine from
> org.hadoop.submarine
> >> to
> >>> org.submarine
> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> >>> 4. On the Github docked travis-ci system, all test cases have been
> >> tested;
> >>> 5. Several Hadoop submarine users completed the system test using the
> >> code
> >>> in this repository.
> >>>
> >>> 赵欣  于2019年7月22日周一 上午9:38写道：
> >>>
> >>>> Hi
> >>>>
> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
> >> are
> >>>> a major in electrical engineering. Our teaching teams and students use
> >>>> bigoop submarine for big data analysis and automation control of
> >>> electrical
> >>>> equipment.
> >>>>
> >>>> Many thanks to the hadoop community for providing us with machine
> >>> learning
> >>>> tools like submarine.
> >>>>
> >>>> I wish hadoop submarine is getting better and better.
> >>>>
> >>>>
> >>>> ==
> >>>> 赵欣
> >>>> 东南大学电气工程学院
> >>>>
> >>>> -
> >>>>
> >>>> Zhao XIN
> >>>>
> >>>> School of Electrical Engineering
> >>>>
> >>>> ==
> >>>> 2019-07-18
> >>>>
> >>>>
> >>>> *From:* Xun Liu 
> >>>> *Date:* 2019-07-18 09:46
> >>>> *To:* xinzhao 
> >>>> *Subject:* Fwd: 

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan]

Thanks everybody for sharing your thoughts. I saw positive feedbacks from
20+ contributors!

So I think we should move it forward, any suggestions about what we should
do?

Best,
Wangda

On Mon, Jul 22, 2019 at 5:36 PM neo  wrote:

> +1, This is neo from TiDB & TiKV community.
> Thanks Xun for bring this up.
>
> Our CNCF project's open source distributed KV storage system TiKV,
> Hadoop submarine's machine learning engine helps us to optimize data
> storage,
> helping us solve some problems in data hotspots and data shuffers.
>
> We are ready to improve the performance of TiDB in our open source
> distributed relational database TiDB and also using the hadoop submarine
> machine learning engine.
>
> I think if submarine can be independent, it will develop faster and better.
> Thanks to the hadoop community for developing submarine!
>
> Best Regards,
> neo
> www.pingcap.com / https://github.com/pingcap/tidb /
> https://github.com/tikv
>
> Xun Liu  于2019年7月22日周一 下午4:07写道：
>
> > @adam.antal
> >
> > The submarine development team has completed the following preparations:
> > 1. Established a temporary test repository on Github.
> > 2. Change the package name of hadoop submarine from org.hadoop.submarine
> to
> > org.submarine
> > 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> > 4. On the Github docked travis-ci system, all test cases have been
> tested;
> > 5. Several Hadoop submarine users completed the system test using the
> code
> > in this repository.
> >
> > 赵欣  于2019年7月22日周一 上午9:38写道：
> >
> > > Hi
> > >
> > > I am a teacher at Southeast University (https://www.seu.edu.cn/). We
> are
> > > a major in electrical engineering. Our teaching teams and students use
> > > bigoop submarine for big data analysis and automation control of
> > electrical
> > > equipment.
> > >
> > > Many thanks to the hadoop community for providing us with machine
> > learning
> > > tools like submarine.
> > >
> > > I wish hadoop submarine is getting better and better.
> > >
> > >
> > > ==
> > > 赵欣
> > > 东南大学电气工程学院
> > >
> > > -
> > >
> > > Zhao XIN
> > >
> > > School of Electrical Engineering
> > >
> > > ==
> > > 2019-07-18
> > >
> > >
> > > *From:* Xun Liu 
> > > *Date:* 2019-07-18 09:46
> > > *To:* xinzhao 
> > > *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> > > project?
> > >
> > >
> > > -- Forwarded message -
> > > 发件人： dashuiguailu...@gmail.com 
> > > Date: 2019年7月17日周三 下午3:17
> > > Subject: Re: Re: Any thoughts making Submarine a separate Apache
> project?
> > > To: Szilard Nemeth , runlin zhang <
> > > runlin...@gmail.com>
> > > Cc: Xun Liu , common-dev <
> > common-...@hadoop.apache.org>,
> > > yarn-dev , hdfs-dev <
> > > hdfs-...@hadoop.apache.org>, mapreduce-dev <
> > > mapreduce-...@hadoop.apache.org>, submarine-dev <
> > > submarine-...@hadoop.apache.org>
> > >
> > >
> > > +1 ，Good idea, we are very much looking forward to it.
> > >
> > > --
> > > dashuiguailu...@gmail.com
> > >
> > >
> > > *From:* Szilard Nemeth 
> > > *Date:* 2019-07-17 14:55
> > > *To:* runlin zhang 
> > > *CC:* Xun Liu ; Hadoop Common
> > > ; yarn-dev ;
> > > Hdfs-dev ; mapreduce-dev
> > > ; submarine-dev
> > > 
> > > *Subject:* Re: Any thoughts making Submarine a separate Apache project?
> > > +1, this is a very great idea.
> > > As Hadoop repository has already grown huge and contains many
> projects, I
> > > think in general it's a good idea to separate projects in the early
> > phase.
> > >
> > >
> > > On Wed, Jul 17, 2019, 08:50 runlin zhang  wrote:
> > >
> > > > +1 ，That will be great ！
> > > >
> > > > > 在 2019年7月10日，下午3:34，Xun Liu  写道：
> > > > >
> > > > > Hi all,
> > > > >
> > > > > This is Xun Liu contributing to the Submarine project for deep
> > learning
> > > > > workloads running with big data workloads together on Hadoop
> > clusters.
> > > > >
> > > > > There are a bunch of integrations of Submarine to other projects
> are
> > > > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> > next
> > > > step
> > > > > of Submarine is going to integrate with more projects like Apache
> > > Arrow,
> > > > > Redis, MLflow, etc. & be able to handle end-to-end machine learning
> > use
> > > > > cases like model serving, notebook management, advanced training
> > > > > optimizations (like auto parameter tuning, memory cache
> optimizations
> > > for
> > > > > large datasets for training, etc.), and make it run on other
> > platforms
> > > > like
> > > > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > > > project
> > > > > to Apache so we can put Submarine and TonY together to the same
> > > codebase
> > > > > (Page #30.
> > > > >
> > > >
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > > > ).
> > > > >
> > > > > This expands the scope of the 

Re: [ANNOUNCE] New Apache Hadoop Committer - Tao Yang

[Wangda Tan]

Congrats!

Best,
Wangda

On Tue, Jul 16, 2019 at 10:37 AM 杨弢(杨弢)  wrote:

> Thanks everyone.
> I'm so honored to be an Apache Hadoop Committer, I will keep working on
> this great project and contribute more. Thanks.
>
> Best Regards,
> Tao Yang
>
>
> --
> 发件人：Naganarasimha Garla 
> 发送时间：2019年7月15日(星期一) 17:55
> 收件人：Weiwei Yang 
> 抄 送：yarn-dev ; Hadoop Common <
> common-...@hadoop.apache.org>; mapreduce-dev <
> mapreduce-...@hadoop.apache.org>; Hdfs-dev 
> 主 题：Re: [ANNOUNCE] New Apache Hadoop Committer - Tao Yang
>
> Congrats and welcome Tao Yang!
>
> Regards
> + Naga
>
> On Mon, 15 Jul 2019, 17:54 Weiwei Yang,  wrote:
>
> > Hi Dear Apache Hadoop Community
> >
> > It's my pleasure to announce that Tao Yang has been elected as an Apache
> > Hadoop committer, this is to recognize his contributions to Apache Hadoop
> > YARN project.
> >
> > Congratulations and welcome on board!
> >
> > Weiwei
> > (On behalf of the Apache Hadoop PMC)
> >
>
>

Re: [Announcement] Apache Hadoop Submarine 0.2.0 released!

[Wangda Tan]

That's great news! Thanks for everybody who helping this release!

Best,
Wangda

On Tue, Jul 2, 2019 at 11:40 PM Zhankun Tang  wrote:

> Hi folks,
>
> I'm glad to announce that the Apache Hadoop community
> has voted to release Apache Hadoop Submarine 0.2.0.
>
> Apache Hadoop Submarine is a project to allows infra engineer/data
> scientist
> to run deep learning applications resource management platform (like YARN,
> K8s).
>
> This release contains below new features:
>
>1. Linkedin’s TonY runtime support in Submarine
>2. PyTorch enabled in Submarine with both YARN native service runtime
>(single node) and TonY runtime
>3. Support uber jar of Submarine to submit the job
>4. The YAML file to describe a job
>5. The Notebook support (by Apache Zeppelin Submarine interpreter)
>
> For more information and to download, please check:
> https://hadoop.apache.org/submarine/
>
> For submarine jar files as project dependencies, please check:
> https://mvnrepository.com/search?q=submarine
>
> Tons of thanks to everyone who contributed to the release, and everyone in
> the Apache Hadoop community! The release is a result of work from many
> contributors. Thank you for all of them.
>
> BR,
> Zhankun
>

Hadoop Community Sync Up Schedule

[Wangda Tan]

Hi folks,

Here's the Hadoop Community Sync Up proposal/schedule:
https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#heading=h.xh4zfwj8ppmn

And here's calendar file:

https://calendar.google.com/calendar/ical/hadoop.community.sync.up%40gmail.com/public/basic.ics

We gave it a try this week for YARN+MR and Submarine sync, feedbacks from
participants seems pretty good, lots of new information shared during sync
up, and companies are using/developing Hadoop can better know each other.

Next week there're 4 community sync-ups (Two Submarine for different
timezones, one YARN+MR, one storage), please join to whichever you're
interested:

[image: image.png]

Zoom info and notes can be found in the Google calendar invitation.

Thanks,
Wangda

Re: Agenda & More Information about Hadoop Community Meetup @ Palo Alto, June 26

[Wangda Tan]

A friendly reminder,

The meetup will take place tomorrow at 9:00 AM PDT to 4:00 PM PDT.

The address is: 395 Page Mill Rd, Palo Alto, CA 94306
We’ll be in the Bigtop conference room on the 1st floor. Go left after
coming through the main entrance, and it will be on the right.

Zoom: https://cloudera.zoom.us/j/606607666

Please let me know if you have any questions. If you haven't RSVP yet,
please go ahead and RSVP so we can better prepare food, seat, etc.

Thanks,
Wangda

On Wed, Jun 19, 2019 at 4:49 PM Wangda Tan  wrote:

> Hi All,
>
> I want to let you know that we have confirmed most of the agenda for
> Hadoop Community Meetup. It will be a whole day event.
>
> Agenda & Dial-In info because see below, *please RSVP
> at https://www.meetup.com/Hadoop-Contributors/events/262055924/
> <https://www.meetup.com/Hadoop-Contributors/events/262055924/>*
>
> Huge thanks to Daniel Templeton, Wei-Chiu Chuang, Christina Vu for helping
> with organizing and logistics.
>
> *Please help to promote meetup information on Twitter, LinkedIn, etc.
> Appreciated! *
>
> Best,
> Wangda
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *AM:9:00: Arrival and check-in--9:30 -
> 10:15:-Talk: Hadoop storage in cloud-native
> environmentsAbstract: Hadoop is a mature storage system but designed years
> before the cloud-native movement. Kubernetes and other cloud-native tools
> are emerging solutions for containerized environments but sometimes they
> require different approaches.In this presentation we would like to share
> our experiences to run Apache Hadoop Ozone in Kubernetes and the connection
> point to other cloud-native ecosystem elements. We will compare the
> benefits and drawbacks to use Kubernetes and Hadoop storage together and
> show our current achievements and future plans.Speaker: Marton Elek
> (Cloudera)10:20 - 11:00:--Talk: Selective Wire Encryption In
> HDFSAbstract: Wire data encryption is a key component of the Hadoop
> Distributed File System (HDFS). However, such encryption enforcement comes
> in as an all-or-nothing feature. In our use case at LinkedIn, we would like
> to selectively expose fast unencrypted access to fully managed internal
> clients, which can be trusted, while only expose encrypted access to
> clients outside of the trusted circle with higher security risks. That way
> we minimize performance overhead for trusted internal clients while still
> securing data from potential outside threats. Our design extends HDFS
> NameNode to run on multiple ports, connecting to different NameNode ports
> would end up with different levels of encryption protection. This
> protection then gets enforced for both NameNode RPC and the subsequent data
> transfers to/from DataNode. This approach comes with minimum operational
> and performance overhead.Speaker: Konstantin Shvachko (LinkedIn), Chen
> Liang (LinkedIn)11:10 - 11:55:-Talk: YuniKorn: Next Generation
> Scheduling for YARN and K8sAbstract: We will talk about our open source
> work - YuniKorn scheduler project (Y for YARN, K for K8s, uni- for Unified)
> brings long-wanted features such as hierarchical queues, fairness between
> users/jobs/queues, preemption to Kubernetes; and it brings service
> scheduling enhancements to YARN. Any improvements to this scheduler can
> benefit both Kubernetes and YARN community.Speaker: Wangda Tan
> (Cloudera)PM:12:00 - 12:55 Lunch Break (Provided by
> Cloudera)1:00 -
> 1:25---Talk: Yarn Efficiency at UberAbstract: We will present the
> work done at Uber to improve YARN cluster utilization and job SOA with
> elastic resource management, low compute workload on passive datacenter,
> preemption, larger container, etc. We will also go through YARN upgrade in
> order to adopt new features and talk about the challenges.Speaker: Aihua Xu
> (Uber), Prashant Golash (Uber)1:30 - 2:10 One more
> talk-2:20 - 4:00---BoF sessions &
> Breakout Sessions & Group discussions: Talk about items like JDK 11
> support, next releases (2.10.0, 3.3.0, etc.), Hadoop on Cloud, etc.4:00:
> Reception provided by
> Cloudera.==Join Zoom
> Meetinghttps://cloudera.zoom.us/j/116816195
> <https://cloudera.zoom.us/j/116816195>*
>

Re: [VOTE] Release Apache Hadoop Submarine 0.2.0 - RC0

[Wangda Tan]

+1 Binding. Tested in local cluster and reviewed docs.

Thanks!

On Wed, Jun 19, 2019 at 3:20 AM Sunil Govindan  wrote:

> +1 binding
>
> - tested in local cluster.
> - tried tony run time as well
> - doc seems fine now.
>
> - Sunil
>
>
> On Thu, Jun 6, 2019 at 6:53 PM Zhankun Tang  wrote:
>
> > Hi folks,
> >
> > Thanks to all of you who have contributed in this submarine 0.2.0
> release.
> > We now have a release candidate (RC0) for Apache Hadoop Submarine 0.2.0.
> >
> >
> > The Artifacts for this Submarine-0.2.0 RC0 are available here:
> >
> > https://home.apache.org/~ztang/submarine-0.2.0-rc0/
> >
> >
> > It's RC tag in git is "submarine-0.2.0-RC0".
> >
> >
> >
> > The maven artifacts are available via repository.apache.org at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1221/
> >
> >
> > This vote will run 7 days (5 weekdays), ending on 13th June at 11:59 pm
> > PST.
> >
> >
> >
> > The highlights of this release.
> >
> > 1. Linkedin's TonY runtime support in Submarine
> >
> > 2. PyTorch enabled in Submarine with both YARN native service runtime
> > (single node) and TonY runtime
> >
> > 3. Support uber jar of Submarine to submit the job
> >
> > 4. The YAML file to describe a job
> >
> > 5. The Notebook support (by Apache Zeppelin Submarine interpreter)
> >
> >
> > Thanks to Sunil, Wangda, Xun, Zac, Keqiu, Szilard for helping me in
> > preparing the release.
> >
> > I have done a few testing with my pseudo cluster. My +1 (non-binding) to
> > start.
> >
> >
> >
> > Regards,
> > Zhankun
> >
>

Agenda & More Information about Hadoop Community Meetup @ Palo Alto, June 26

[Wangda Tan]

Hi All,

I want to let you know that we have confirmed most of the agenda for Hadoop
Community Meetup. It will be a whole day event.

Agenda & Dial-In info because see below, *please RSVP
at https://www.meetup.com/Hadoop-Contributors/events/262055924/
<https://www.meetup.com/Hadoop-Contributors/events/262055924/>*

Huge thanks to Daniel Templeton, Wei-Chiu Chuang, Christina Vu for helping
with organizing and logistics.

*Please help to promote meetup information on Twitter, LinkedIn, etc.
Appreciated! *

Best,
Wangda

























































*AM:9:00: Arrival and check-in--9:30 -
10:15:-Talk: Hadoop storage in cloud-native
environmentsAbstract: Hadoop is a mature storage system but designed years
before the cloud-native movement. Kubernetes and other cloud-native tools
are emerging solutions for containerized environments but sometimes they
require different approaches.In this presentation we would like to share
our experiences to run Apache Hadoop Ozone in Kubernetes and the connection
point to other cloud-native ecosystem elements. We will compare the
benefits and drawbacks to use Kubernetes and Hadoop storage together and
show our current achievements and future plans.Speaker: Marton Elek
(Cloudera)10:20 - 11:00:--Talk: Selective Wire Encryption In
HDFSAbstract: Wire data encryption is a key component of the Hadoop
Distributed File System (HDFS). However, such encryption enforcement comes
in as an all-or-nothing feature. In our use case at LinkedIn, we would like
to selectively expose fast unencrypted access to fully managed internal
clients, which can be trusted, while only expose encrypted access to
clients outside of the trusted circle with higher security risks. That way
we minimize performance overhead for trusted internal clients while still
securing data from potential outside threats. Our design extends HDFS
NameNode to run on multiple ports, connecting to different NameNode ports
would end up with different levels of encryption protection. This
protection then gets enforced for both NameNode RPC and the subsequent data
transfers to/from DataNode. This approach comes with minimum operational
and performance overhead.Speaker: Konstantin Shvachko (LinkedIn), Chen
Liang (LinkedIn)11:10 - 11:55:-Talk: YuniKorn: Next Generation
Scheduling for YARN and K8sAbstract: We will talk about our open source
work - YuniKorn scheduler project (Y for YARN, K for K8s, uni- for Unified)
brings long-wanted features such as hierarchical queues, fairness between
users/jobs/queues, preemption to Kubernetes; and it brings service
scheduling enhancements to YARN. Any improvements to this scheduler can
benefit both Kubernetes and YARN community.Speaker: Wangda Tan
(Cloudera)PM:12:00 - 12:55 Lunch Break (Provided by
Cloudera)1:00 -
1:25---Talk: Yarn Efficiency at UberAbstract: We will present the
work done at Uber to improve YARN cluster utilization and job SOA with
elastic resource management, low compute workload on passive datacenter,
preemption, larger container, etc. We will also go through YARN upgrade in
order to adopt new features and talk about the challenges.Speaker: Aihua Xu
(Uber), Prashant Golash (Uber)1:30 - 2:10 One more
talk-2:20 - 4:00---BoF sessions &
Breakout Sessions & Group discussions: Talk about items like JDK 11
support, next releases (2.10.0, 3.3.0, etc.), Hadoop on Cloud, etc.4:00:
Reception provided by
Cloudera.==Join Zoom
Meetinghttps://cloudera.zoom.us/j/116816195
<https://cloudera.zoom.us/j/116816195>*

Re: [DISCUSS] A unified and open Hadoop community sync up schedule?

[Wangda Tan]

Thanks @Wei-Chiu Chuang  . updated gdoc

On Tue, Jun 18, 2019 at 7:35 PM Wei-Chiu Chuang  wrote:

> Thanks Wangda,
>
> I just like to make a correction -- the .ics calendar file says the first
> Wednesday for HDFS/cloud connector is in Mandarin whereas on the gdoc is to
> host it on the third Wednesday.
>
> On Tue, Jun 18, 2019 at 5:29 PM Wangda Tan  wrote:
>
> > Hi Folks,
> >
> > I just updated doc:
> >
> >
> https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#
> > with
> > dial-in information, notes, etc.
> >
> > Here's a calendar to subscribe:
> >
> >
> https://calendar.google.com/calendar/ical/hadoop.community.sync.up%40gmail.com/public/basic.ics
> >
> > I'm thinking to give it a try from next week, any suggestions?
> >
> > Thanks,
> > Wangda
> >
> > On Fri, Jun 14, 2019 at 4:02 PM Wangda Tan  wrote:
> >
> > > And please let me know if you can help with coordinate logistics stuff,
> > > cross-checking, etc. Let's spend some time next week to get it
> finalized.
> > >
> > > Thanks,
> > > Wangda
> > >
> > > On Fri, Jun 14, 2019 at 4:00 PM Wangda Tan 
> wrote:
> > >
> > >> Hi Folks,
> > >>
> > >> Yufei: Agree with all your opinions.
> > >>
> > >> Anu: it might be more efficient to use Google doc to track meeting
> > >> minutes and we can put them together.
> > >>
> > >> I just put the proposal to
> > >>
> >
> https://calendar.google.com/calendar/b/3?cid=aGFkb29wLmNvbW11bml0eS5zeW5jLnVwQGdtYWlsLmNvbQ
> > ,
> > >> you can check if the proposal time works or not. If you agree, we can
> go
> > >> ahead to add meeting link, google doc, etc.
> > >>
> > >> If you want to have edit permissions, please drop a private email to
> me
> > >> so I will add you.
> > >>
> > >> We still need more hosts, in each track, ideally we should have at
> least
> > >> 3 hosts per track just like HDFS blocks :), please volunteer, so we
> can
> > >> have enough members to run the meeting.
> > >>
> > >> Let's shoot by end of the next week, let's get all logistics done and
> > >> starting community sync up series from the week of Jun 25th.
> > >>
> > >> Thanks,
> > >> Wangda
> > >>
> > >> Thanks,
> > >> Wangda
> > >>
> > >>
> > >>
> > >> On Tue, Jun 11, 2019 at 10:23 AM Anu Engineer  >
> > >> wrote:
> > >>
> > >>> For Ozone, we have started using the Wiki itself as the agenda and
> > after
> > >>> the meeting is over, we convert it into the meeting notes.
> > >>> Here is an example, the project owner can edit and maintain it, it is
> > >>> like 10 mins work - and allows anyone to add stuff into the agenda
> too.
> > >>>
> > >>>
> > >>>
> >
> https://cwiki.apache.org/confluence/display/HADOOP/2019-06-10+Meeting+notes
> > >>>
> > >>> --Anu
> > >>>
> > >>> On Tue, Jun 11, 2019 at 10:20 AM Yufei Gu 
> > wrote:
> > >>>
> > >>>> +1 for this idea. Thanks Wangda for bringing this up.
> > >>>>
> > >>>> Some comments to share:
> > >>>>
> > >>>>- Agenda needed to be posted ahead of meeting and welcome any
> > >>>> interested
> > >>>>party to contribute to topics.
> > >>>>- We should encourage more people to attend. That's whole point
> of
> > >>>> the
> > >>>>meeting.
> > >>>>- Hopefully, this can mitigate the situation that some patches
> are
> > >>>>waiting for review for ever, which turns away new contributors.
> > >>>>- 30m per session sounds a little bit short, we can try it out
> and
> > >>>> see
> > >>>>if extension is needed.
> > >>>>
> > >>>> Best,
> > >>>>
> > >>>> Yufei
> > >>>>
> > >>>> `This is not a contribution`
> > >>>>
> > >>>>
> > >>>> On Fri, Jun 7, 2019 at 4:39 PM Wangda Tan 
> > wrote:
> > >>>>
> > >>>> > Hi Hadoop-devs,
> > &

Re: [DISCUSS] A unified and open Hadoop community sync up schedule?

[Wangda Tan]

Hi Folks,

I just updated doc:
https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#
with
dial-in information, notes, etc.

Here's a calendar to subscribe:
https://calendar.google.com/calendar/ical/hadoop.community.sync.up%40gmail.com/public/basic.ics

I'm thinking to give it a try from next week, any suggestions?

Thanks,
Wangda

On Fri, Jun 14, 2019 at 4:02 PM Wangda Tan  wrote:

> And please let me know if you can help with coordinate logistics stuff,
> cross-checking, etc. Let's spend some time next week to get it finalized.
>
> Thanks,
> Wangda
>
> On Fri, Jun 14, 2019 at 4:00 PM Wangda Tan  wrote:
>
>> Hi Folks,
>>
>> Yufei: Agree with all your opinions.
>>
>> Anu: it might be more efficient to use Google doc to track meeting
>> minutes and we can put them together.
>>
>> I just put the proposal to
>> https://calendar.google.com/calendar/b/3?cid=aGFkb29wLmNvbW11bml0eS5zeW5jLnVwQGdtYWlsLmNvbQ,
>> you can check if the proposal time works or not. If you agree, we can go
>> ahead to add meeting link, google doc, etc.
>>
>> If you want to have edit permissions, please drop a private email to me
>> so I will add you.
>>
>> We still need more hosts, in each track, ideally we should have at least
>> 3 hosts per track just like HDFS blocks :), please volunteer, so we can
>> have enough members to run the meeting.
>>
>> Let's shoot by end of the next week, let's get all logistics done and
>> starting community sync up series from the week of Jun 25th.
>>
>> Thanks,
>> Wangda
>>
>> Thanks,
>> Wangda
>>
>>
>>
>> On Tue, Jun 11, 2019 at 10:23 AM Anu Engineer 
>> wrote:
>>
>>> For Ozone, we have started using the Wiki itself as the agenda and after
>>> the meeting is over, we convert it into the meeting notes.
>>> Here is an example, the project owner can edit and maintain it, it is
>>> like 10 mins work - and allows anyone to add stuff into the agenda too.
>>>
>>>
>>> https://cwiki.apache.org/confluence/display/HADOOP/2019-06-10+Meeting+notes
>>>
>>> --Anu
>>>
>>> On Tue, Jun 11, 2019 at 10:20 AM Yufei Gu  wrote:
>>>
>>>> +1 for this idea. Thanks Wangda for bringing this up.
>>>>
>>>> Some comments to share:
>>>>
>>>>- Agenda needed to be posted ahead of meeting and welcome any
>>>> interested
>>>>party to contribute to topics.
>>>>- We should encourage more people to attend. That's whole point of
>>>> the
>>>>meeting.
>>>>- Hopefully, this can mitigate the situation that some patches are
>>>>waiting for review for ever, which turns away new contributors.
>>>>- 30m per session sounds a little bit short, we can try it out and
>>>> see
>>>>if extension is needed.
>>>>
>>>> Best,
>>>>
>>>> Yufei
>>>>
>>>> `This is not a contribution`
>>>>
>>>>
>>>> On Fri, Jun 7, 2019 at 4:39 PM Wangda Tan  wrote:
>>>>
>>>> > Hi Hadoop-devs,
>>>> >
>>>> > Previous we have regular YARN community sync up (1 hr, biweekly, but
>>>> not
>>>> > open to public). Recently because of changes in our schedules, Less
>>>> folks
>>>> > showed up in the sync up for the last several months.
>>>> >
>>>> > I saw the K8s community did a pretty good job to run their sig
>>>> meetings,
>>>> > there's regular meetings for different topics, notes, agenda, etc.
>>>> Such as
>>>> >
>>>> >
>>>> https://docs.google.com/document/d/13mwye7nvrmV11q9_Eg77z-1w3X7Q1GTbslpml4J7F3A/edit
>>>> >
>>>> >
>>>> > For Hadoop community, there are less such regular meetings open to the
>>>> > public except for Ozone project and offline meetups or
>>>> Bird-of-Features in
>>>> > Hadoop/DataWorks Summit. Recently we have a few folks joined DataWorks
>>>> > Summit at Washington DC and Barcelona, and lots (50+) of folks join
>>>> the
>>>> > Ozone/Hadoop/YARN BoF, ask (good) questions and roadmaps. I think it
>>>> is
>>>> > important to open such conversations to the public and let more
>>>> > folk/companies join.
>>>> >
>>>> > Discussed a small group of community members and wrote a short
>>>> proposal
>>>> > about the form, time and topic of the community sync up, thanks for
>>>> > everybody who have contributed to the proposal! Please feel free to
>>>> add
>>>> > your thoughts to the Proposal Google doc
>>>> > <
>>>> >
>>>> https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#
>>>> > >
>>>> > .
>>>> >
>>>> > Especially for the following parts:
>>>> > - If you have interests to run any of the community sync-ups, please
>>>> put
>>>> > your name to the table inside the proposal. We need more volunteers
>>>> to help
>>>> > run the sync-ups in different timezones.
>>>> > - Please add suggestions to the time, frequency and themes and feel
>>>> free to
>>>> > share your thoughts if we should do sync ups for other topics which
>>>> are not
>>>> > covered by the proposal.
>>>> >
>>>> > Link to the Proposal Google doc
>>>> > <
>>>> >
>>>> https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#
>>>> > >
>>>> >
>>>> > Thanks,
>>>> > Wangda Tan
>>>> >
>>>>
>>>

Re: [DISCUSS] A unified and open Hadoop community sync up schedule?

[Wangda Tan]

And please let me know if you can help with coordinate logistics stuff,
cross-checking, etc. Let's spend some time next week to get it finalized.

Thanks,
Wangda

On Fri, Jun 14, 2019 at 4:00 PM Wangda Tan  wrote:

> Hi Folks,
>
> Yufei: Agree with all your opinions.
>
> Anu: it might be more efficient to use Google doc to track meeting minutes
> and we can put them together.
>
> I just put the proposal to
> https://calendar.google.com/calendar/b/3?cid=aGFkb29wLmNvbW11bml0eS5zeW5jLnVwQGdtYWlsLmNvbQ,
> you can check if the proposal time works or not. If you agree, we can go
> ahead to add meeting link, google doc, etc.
>
> If you want to have edit permissions, please drop a private email to me so
> I will add you.
>
> We still need more hosts, in each track, ideally we should have at least 3
> hosts per track just like HDFS blocks :), please volunteer, so we can have
> enough members to run the meeting.
>
> Let's shoot by end of the next week, let's get all logistics done and
> starting community sync up series from the week of Jun 25th.
>
> Thanks,
> Wangda
>
> Thanks,
> Wangda
>
>
>
> On Tue, Jun 11, 2019 at 10:23 AM Anu Engineer 
> wrote:
>
>> For Ozone, we have started using the Wiki itself as the agenda and after
>> the meeting is over, we convert it into the meeting notes.
>> Here is an example, the project owner can edit and maintain it, it is
>> like 10 mins work - and allows anyone to add stuff into the agenda too.
>>
>>
>> https://cwiki.apache.org/confluence/display/HADOOP/2019-06-10+Meeting+notes
>>
>> --Anu
>>
>> On Tue, Jun 11, 2019 at 10:20 AM Yufei Gu  wrote:
>>
>>> +1 for this idea. Thanks Wangda for bringing this up.
>>>
>>> Some comments to share:
>>>
>>>- Agenda needed to be posted ahead of meeting and welcome any
>>> interested
>>>party to contribute to topics.
>>>- We should encourage more people to attend. That's whole point of the
>>>meeting.
>>>- Hopefully, this can mitigate the situation that some patches are
>>>waiting for review for ever, which turns away new contributors.
>>>- 30m per session sounds a little bit short, we can try it out and see
>>>if extension is needed.
>>>
>>> Best,
>>>
>>> Yufei
>>>
>>> `This is not a contribution`
>>>
>>>
>>> On Fri, Jun 7, 2019 at 4:39 PM Wangda Tan  wrote:
>>>
>>> > Hi Hadoop-devs,
>>> >
>>> > Previous we have regular YARN community sync up (1 hr, biweekly, but
>>> not
>>> > open to public). Recently because of changes in our schedules, Less
>>> folks
>>> > showed up in the sync up for the last several months.
>>> >
>>> > I saw the K8s community did a pretty good job to run their sig
>>> meetings,
>>> > there's regular meetings for different topics, notes, agenda, etc.
>>> Such as
>>> >
>>> >
>>> https://docs.google.com/document/d/13mwye7nvrmV11q9_Eg77z-1w3X7Q1GTbslpml4J7F3A/edit
>>> >
>>> >
>>> > For Hadoop community, there are less such regular meetings open to the
>>> > public except for Ozone project and offline meetups or
>>> Bird-of-Features in
>>> > Hadoop/DataWorks Summit. Recently we have a few folks joined DataWorks
>>> > Summit at Washington DC and Barcelona, and lots (50+) of folks join the
>>> > Ozone/Hadoop/YARN BoF, ask (good) questions and roadmaps. I think it is
>>> > important to open such conversations to the public and let more
>>> > folk/companies join.
>>> >
>>> > Discussed a small group of community members and wrote a short proposal
>>> > about the form, time and topic of the community sync up, thanks for
>>> > everybody who have contributed to the proposal! Please feel free to add
>>> > your thoughts to the Proposal Google doc
>>> > <
>>> >
>>> https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#
>>> > >
>>> > .
>>> >
>>> > Especially for the following parts:
>>> > - If you have interests to run any of the community sync-ups, please
>>> put
>>> > your name to the table inside the proposal. We need more volunteers to
>>> help
>>> > run the sync-ups in different timezones.
>>> > - Please add suggestions to the time, frequency and themes and feel
>>> free to
>>> > share your thoughts if we should do sync ups for other topics which
>>> are not
>>> > covered by the proposal.
>>> >
>>> > Link to the Proposal Google doc
>>> > <
>>> >
>>> https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#
>>> > >
>>> >
>>> > Thanks,
>>> > Wangda Tan
>>> >
>>>
>>

Re: [DISCUSS] A unified and open Hadoop community sync up schedule?

[Wangda Tan]

Hi Folks,

Yufei: Agree with all your opinions.

Anu: it might be more efficient to use Google doc to track meeting minutes
and we can put them together.

I just put the proposal to
https://calendar.google.com/calendar/b/3?cid=aGFkb29wLmNvbW11bml0eS5zeW5jLnVwQGdtYWlsLmNvbQ,
you can check if the proposal time works or not. If you agree, we can go
ahead to add meeting link, google doc, etc.

If you want to have edit permissions, please drop a private email to me so
I will add you.

We still need more hosts, in each track, ideally we should have at least 3
hosts per track just like HDFS blocks :), please volunteer, so we can have
enough members to run the meeting.

Let's shoot by end of the next week, let's get all logistics done and
starting community sync up series from the week of Jun 25th.

Thanks,
Wangda

Thanks,
Wangda



On Tue, Jun 11, 2019 at 10:23 AM Anu Engineer 
wrote:

> For Ozone, we have started using the Wiki itself as the agenda and after
> the meeting is over, we convert it into the meeting notes.
> Here is an example, the project owner can edit and maintain it, it is like
> 10 mins work - and allows anyone to add stuff into the agenda too.
>
> https://cwiki.apache.org/confluence/display/HADOOP/2019-06-10+Meeting+notes
>
> --Anu
>
> On Tue, Jun 11, 2019 at 10:20 AM Yufei Gu  wrote:
>
>> +1 for this idea. Thanks Wangda for bringing this up.
>>
>> Some comments to share:
>>
>>- Agenda needed to be posted ahead of meeting and welcome any
>> interested
>>party to contribute to topics.
>>- We should encourage more people to attend. That's whole point of the
>>meeting.
>>- Hopefully, this can mitigate the situation that some patches are
>>waiting for review for ever, which turns away new contributors.
>>- 30m per session sounds a little bit short, we can try it out and see
>>if extension is needed.
>>
>> Best,
>>
>> Yufei
>>
>> `This is not a contribution`
>>
>>
>> On Fri, Jun 7, 2019 at 4:39 PM Wangda Tan  wrote:
>>
>> > Hi Hadoop-devs,
>> >
>> > Previous we have regular YARN community sync up (1 hr, biweekly, but not
>> > open to public). Recently because of changes in our schedules, Less
>> folks
>> > showed up in the sync up for the last several months.
>> >
>> > I saw the K8s community did a pretty good job to run their sig meetings,
>> > there's regular meetings for different topics, notes, agenda, etc. Such
>> as
>> >
>> >
>> https://docs.google.com/document/d/13mwye7nvrmV11q9_Eg77z-1w3X7Q1GTbslpml4J7F3A/edit
>> >
>> >
>> > For Hadoop community, there are less such regular meetings open to the
>> > public except for Ozone project and offline meetups or Bird-of-Features
>> in
>> > Hadoop/DataWorks Summit. Recently we have a few folks joined DataWorks
>> > Summit at Washington DC and Barcelona, and lots (50+) of folks join the
>> > Ozone/Hadoop/YARN BoF, ask (good) questions and roadmaps. I think it is
>> > important to open such conversations to the public and let more
>> > folk/companies join.
>> >
>> > Discussed a small group of community members and wrote a short proposal
>> > about the form, time and topic of the community sync up, thanks for
>> > everybody who have contributed to the proposal! Please feel free to add
>> > your thoughts to the Proposal Google doc
>> > <
>> >
>> https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#
>> > >
>> > .
>> >
>> > Especially for the following parts:
>> > - If you have interests to run any of the community sync-ups, please put
>> > your name to the table inside the proposal. We need more volunteers to
>> help
>> > run the sync-ups in different timezones.
>> > - Please add suggestions to the time, frequency and themes and feel
>> free to
>> > share your thoughts if we should do sync ups for other topics which are
>> not
>> > covered by the proposal.
>> >
>> > Link to the Proposal Google doc
>> > <
>> >
>> https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#
>> > >
>> >
>> > Thanks,
>> > Wangda Tan
>> >
>>
>

[DISCUSS] A unified and open Hadoop community sync up schedule?

[Wangda Tan]

Hi Hadoop-devs,

Previous we have regular YARN community sync up (1 hr, biweekly, but not
open to public). Recently because of changes in our schedules, Less folks
showed up in the sync up for the last several months.

I saw the K8s community did a pretty good job to run their sig meetings,
there's regular meetings for different topics, notes, agenda, etc. Such as
https://docs.google.com/document/d/13mwye7nvrmV11q9_Eg77z-1w3X7Q1GTbslpml4J7F3A/edit


For Hadoop community, there are less such regular meetings open to the
public except for Ozone project and offline meetups or Bird-of-Features in
Hadoop/DataWorks Summit. Recently we have a few folks joined DataWorks
Summit at Washington DC and Barcelona, and lots (50+) of folks join the
Ozone/Hadoop/YARN BoF, ask (good) questions and roadmaps. I think it is
important to open such conversations to the public and let more
folk/companies join.

Discussed a small group of community members and wrote a short proposal
about the form, time and topic of the community sync up, thanks for
everybody who have contributed to the proposal! Please feel free to add
your thoughts to the Proposal Google doc
<https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#>
.

Especially for the following parts:
- If you have interests to run any of the community sync-ups, please put
your name to the table inside the proposal. We need more volunteers to help
run the sync-ups in different timezones.
- Please add suggestions to the time, frequency and themes and feel free to
share your thoughts if we should do sync ups for other topics which are not
covered by the proposal.

Link to the Proposal Google doc
<https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#>

Thanks,
Wangda Tan

Re: [VOTE] Release Apache Hadoop Submarine 0.2.0 - RC0

[Wangda Tan]

Thanks Zhankun and everybody who helped this release.

Could you share the link to the user doc?

Best,
Wangda

On Thu, Jun 6, 2019 at 8:55 AM Xun Liu  wrote:

> Zhankun,
>
> Thank you release submarine 0.2.0
>
> I tested it with submarine-0.2.0-rc0. The functions can be used normally.
> But I found a little problem, The command parameter --verbose does not
> take effect.
> But this does not affect the use, It is recommended that after
> submarine-0.2.0-rc0 is released, Can be improved.
>
> +1
>
>
> > On Jun 6, 2019, at 9:23 PM, Zhankun Tang  wrote:
> >
> > Hi folks,
> >
> > Thanks to all of you who have contributed in this submarine 0.2.0
> release.
> > We now have a release candidate (RC0) for Apache Hadoop Submarine 0.2.0.
> >
> >
> > The Artifacts for this Submarine-0.2.0 RC0 are available here:
> >
> > https://home.apache.org/~ztang/submarine-0.2.0-rc0/
> >
> >
> > It's RC tag in git is "submarine-0.2.0-RC0".
> >
> >
> >
> > The maven artifacts are available via repository.apache.org at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1221/
> >
> >
> > This vote will run 7 days (5 weekdays), ending on 13th June at 11:59 pm
> PST.
> >
> >
> >
> > The highlights of this release.
> >
> > 1. Linkedin's TonY runtime support in Submarine
> >
> > 2. PyTorch enabled in Submarine with both YARN native service runtime
> > (single node) and TonY runtime
> >
> > 3. Support uber jar of Submarine to submit the job
> >
> > 4. The YAML file to describe a job
> >
> > 5. The Notebook support (by Apache Zeppelin Submarine interpreter)
> >
> >
> > Thanks to Sunil, Wangda, Xun, Zac, Keqiu, Szilard for helping me in
> > preparing the release.
> >
> > I have done a few testing with my pseudo cluster. My +1 (non-binding) to
> > start.
> >
> >
> >
> > Regards,
> > Zhankun
>
>
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: [ANNOUNCE] Eric Badger is now a committer!

[Wangda Tan]

Congratulations, Eric.

Welcome aboard!

Best,
Wangda


On Tue, Mar 5, 2019 at 2:26 PM Sree V 
wrote:

> Congratulations, Eric.
>
>
>
> Thank you./Sree
>
>
>
> On Tuesday, March 5, 2019, 12:50:20 PM PST, Ayush Saxena <
> ayush...@gmail.com> wrote:
>
>  Congratulations Eric!!!
>
> -Ayush
>
> > On 05-Mar-2019, at 11:34 PM, Chandni Singh 
> wrote:
> >
> > Congratulations Eric!
> >
> > On Tue, Mar 5, 2019 at 9:32 AM Jim Brennan
> >  wrote:
> >
> >> Congratulations Eric!
> >>
> >> On Tue, Mar 5, 2019 at 11:20 AM Eric Payne  >> .invalid>
> >> wrote:
> >>
> >>> It is my pleasure to announce that Eric Badger has accepted an
> invitation
> >>> to become a Hadoop Core committer.
> >>>
> >>> Congratulations, Eric! This is well-deserved!
> >>>
> >>> -Eric Payne
> >>>
> >>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: Submarine: How to contribute?

[Wangda Tan]

Thanks Xun of this answer, I just added info to Submarine Wiki about email
list, etc.

https://cwiki.apache.org/confluence/display/HADOOP/Submarine+Wiki+Resources

Please move dev-related discussions to submarine-dev list.

Thanks,
Wangda

On Wed, Feb 27, 2019 at 7:43 PM Xun Liu  wrote:

> Hi, Kshiteej Mahajan
> Thank you for interest in the hadoop submarine project. :-)
> The submarine pause and submarine resume functions in the submarine design
> documentation have not been developed and implemented.
> You can contribute your code by following these steps:
> You can first create a JIRA on
> https://issues.apache.org/jira/projects/SUBMARINE <
> https://issues.apache.org/jira/projects/SUBMARINE> about the function you
> want to implement. For example, the JIRA number is: SUBMARINE-XXX.
> Copy the code: git clone https://github.com/apache/hadoop.git
> Create a development branch: git checkout -b SUBMARINE-XXX
> You can add code for submarine pause and submarine resume in
> /apache/hadoop/tree/trunk/hadoop-submarine/hadoop-submarine-core to
> complete code development and testing.
> Create a patch: merge your new code into a patch file
> Remove the patch header information: delete the first 5 lines in the patch
> file
> Submit the patch: At https://issues.apache.org/jira/browse/SUBMARINE-XXX <
> https://issues.apache.org/jira/browse/SUBMARINE-XXX>, click on the
> [submit patch] button to upload your patch file.
> Wait for jenkins to compile: Each time you submit a patch file, it will
> trigger jenkins to compile your code and execute all the test cases in
> hadoop.
> Compile and test pass: There will be a committer accepting the patch file
> you submitted, your code will be merged into the hadoop branch.
> Boldly submit your code, good luck!
>
> > 在 2019年2月28日，上午7:18，Kshiteej Mahajan  写道：
> >
> > Hi Devs,
> >
> > I had a few questions about the Submarine project. I browsed through the
> > Submarine design doc:
> >
> https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit
> > and
> > it says that it supports API functions like "submarine pause" and
> > "submarine resume". However, when I browsed through the source code I
> could
> > not find support for these API's (maybe I looked in the wrong places).
> >
> > Are these API features or any new API features for Submarine still under
> > development? I am interested in contributing to these efforts. Please,
> can
> > anyone point me in the right direction as to how to contribute?
> >
> > Thanks,
> > Kshiteej Mahajan
> > Graduate Student at UW-Madison
>
>

Re: [VOTE] Propose to start new Hadoop sub project "submarine"

[Wangda Tan]

Thanks everybody for voting!

We got 10 binding votes and 15 non-binding votes. The vote passed. We will
work with the community to start new subproject.

Best,
Wangda

On Mon, Feb 4, 2019 at 9:06 PM Prabhu Josephraj 
wrote:

> +1 (non-binding)
>
> On Tue, Feb 5, 2019 at 10:30 AM Naganarasimha Garla <
> naganarasimha...@apache.org> wrote:
>
>> +1
>>
>> On Sat, 2 Feb 2019, 09:51 Rohith Sharma K S > wrote:
>>
>> > +1
>> >
>> > On Sat, Feb 2, 2019, 3:54 AM Wangda Tan  wrote:
>> >
>> > > Hi all,
>> > >
>> > > According to positive feedbacks from the thread [1]
>> > >
>> > > This is vote thread to start a new subproject named "hadoop-submarine"
>> > > which follows the release process already established for ozone.
>> > >
>> > > The vote runs for usual 7 days, which ends at Feb 8th 5 PM PDT.
>> > >
>> > > Thanks,
>> > > Wangda Tan
>> > >
>> > > [1]
>> > >
>> > >
>> >
>> https://lists.apache.org/thread.html/f864461eb188bd12859d51b0098ec38942c4429aae7e4d001a633d96@%3Cyarn-dev.hadoop.apache.org%3E
>> > >
>> >
>>
>

[ANNOUNCE] Apache Hadoop 3.1.2 release

[Wangda Tan]

It gives us great pleasure to announce that the Apache Hadoop community has
voted to release Apache Hadoop 3.1.2.

IMPORTANT NOTES

3.1.2 is the second stable release of 3.1 line which is considered to be
production-ready.

Hadoop 3.1.2 brings a number of enhancements.

The Hadoop community fixed 325 JIRAs [1] in total as part of the 3.1.2
release. Of these fixes:

Apache Hadoop 3.1.2 contains a number of significant features and
enhancements. A few of them are noted below.

- Nvidia-docker-plugin v2 support for GPU support on YARN.
- YARN service upgrade improvements and bug fixes.
- YARN UIv2 improvements and bug fixes.
- AliyunOSS related improvements and bug fixes.
- Docker on YARN support related improvements and bug fixes.

Please see the Hadoop 3.1.2 CHANGES for the detailed list of issues
resolved. The release news is posted on the Apache Hadoop website too, you
can go to the downloads section.

Many thanks to everyone who contributed to the release, and everyone in the
Apache Hadoop community! The release is a result of direct and indirect
efforts from many contributors, listed below are the those who contributed
directly by submitting patches and/or reporting issues. (148 contributors,
Sorted by ID)

BilwaST, Charo Zhang, GeLiXin, Harsha1206, Huachao, Jim_Brennan, LiJinglun,
Naganarasimha, OrDTesters, RANith, Rakesh_Shah, Ray Burgemeestre, Sen Zhao,
SoumyaPN, SouryakantaDwivedy, Tao Yang, Zian Chen, abmodi, adam.antal,
ajayydv, ajisakaa, akhilpb, akhilsnaik, amihalyi, arpitagarwal, aw,
ayushtkn, banditka, belugabehr, benlau, bibinchundatt, billie.rinaldi,
boky01, bolke, borisvu, botong, brahmareddy, briandburton, bsteinbach,
candychencan, ccondit-target, charanh, cheersyang, cltlfcjin, collinma,
crh, csingh, csun, daisuke.kobayashi, daryn, dibyendu_hadoop,
dineshchitlangia, ebadger, eepayne, elgoiri, erwaman, eyang, fengchuang,
ferhui, fly_in_gis, gabor.bota, gezapeti, gsaha, haibochen, hexiaoqiao,
hfyang20071, hgadre, jeagles, jhung, jiangjianfei, jianliang.wu,
jira.shegalov, jiwq, jlowe, jojochuang, jonBoone, kanwaljeets, karams,
kennethlnnn, kgyrtkirk, kihwal, knanasi, kshukla, laszlok, leftnoteasy,
leiqiang, liaoyuxiangqin, linyiqun, ljain, lukmajercak, maniraj...@gmail.com,
masatana, nandakumar131, oliverhuh...@gmail.com, oshevchenko, pbacsko,
peruguusha, photogamrun, pj.fanning, prabham, pradeepambati, pranay_singh,
revans2, rkanter, rohithsharma, shaneku...@gmail.com, shubham.dewan,
shuzirra, shv, simonprewo, sinago, smeng, snemeth, sodonnell,
sreenivasulureddy, ssath...@hortonworks.com, ssulav, ste...@apache.org,
study, suma.shivaprasad, sunilg, surendrasingh, tangzhankun, tarunparimi,
tasanuma0829, templedf, thinktaocs, tlipcon, tmarquardt, trjianjianjiao,
uranus, varun_saxena, vinayrpet, vrushalic, wilfreds,
write2kish...@gmail.com, wujinhu, xiaochen, xiaoheipangzi, xkrogen,
yangjiandan, yeshavora, yiran, yoelee, yuzhih...@gmail.com, zichensun,
zvenczel

Wangda Tan and Sunil Govind

[1] JIRA query: project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution =
Fixed AND fixVersion = 3.1.2 ORDER BY key ASC, updated ASC, created DESC,
priority DESC

Re: [VOTE] Moving branch-2 precommit/nightly test builds to java 8

[Wangda Tan]

+1, make sense to me.

On Tue, Feb 5, 2019 at 3:29 PM Konstantin Shvachko 
wrote:

> +1 Makes sense to me.
>
> Thanks,
> --Konst
>
> On Mon, Feb 4, 2019 at 6:14 PM Jonathan Hung  wrote:
>
> > Hello,
> >
> > Starting a vote based on the discuss thread [1] for moving branch-2
> > precommit/nightly test builds to openjdk8. After this change, the test
> > phase for precommit builds [2] and branch-2 nightly build [3] will run on
> > openjdk8. To maintain source compatibility, these builds will still run
> > their compile phase for branch-2 on openjdk7 as they do now (in addition
> to
> > compiling on openjdk8).
> >
> > Vote will run for three business days until Thursday Feb 7 6:00PM PDT.
> >
> > [1]
> >
> >
> https://lists.apache.org/thread.html/7e6fb28fc67560f83a2eb62752df35a8d58d86b2a3df4cacb5d738ca@%3Ccommon-dev.hadoop.apache.org%3E
> >
> > [2]
> >
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HADOOP-Build/
> > https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HDFS-Build/
> > https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-YARN-Build/
> >
> >
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/
> >
> > [3]
> >
> >
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
> >
> > Jonathan Hung
> >
>

Re: [VOTE] Release Apache Hadoop 3.1.2 - RC1

[Wangda Tan]

Thanks everyone for voting, we have 6 binding votes and 6 non-binding
votes, no vetos. The vote passed. We will go ahead to stage this release.

- Wangda and Sunil.

On Tue, Feb 5, 2019 at 12:05 PM Billie Rinaldi 
wrote:

> Thanks Sunil, the new source tarball matches the RC tag and its checksum
> and signature look good.
>
> Billie
>
> On Tue, Feb 5, 2019 at 10:50 AM Sunil G  wrote:
>
> > Thanks Billie for pointing out.
> > I have updated source by removing patchprocess and extra line create
> > release.
> >
> > Also updated checksum as well.
> >
> > @bil...@apache.org   @Wangda Tan 
> > please help to verify this changed bit once.
> >
> > Thanks
> > Sunil
> >
> > On Tue, Feb 5, 2019 at 5:23 AM Billie Rinaldi 
> > wrote:
> >
> >> Hey Sunil and Wangda, thanks for the RC. The source tarball has a
> >> patchprocess directory with some yetus code in it. Also, the file
> >> dev-support/bin/create-release file has the following line added:
> >>   export GPG_AGENT_INFO="/home/sunilg/.gnupg/S.gpg-agent:$(pgrep
> >> gpg-agent):1"
> >>
> >> I think we are probably due for an overall review of LICENSE and NOTICE.
> >> I saw some idiosyncrasies there but nothing that looked like a blocker.
> >>
> >> On Mon, Jan 28, 2019 at 10:20 PM Sunil G  wrote:
> >>
> >>> Hi Folks,
> >>>
> >>> On behalf of Wangda, we have an RC1 for Apache Hadoop 3.1.2.
> >>>
> >>> The artifacts are available here:
> >>> http://home.apache.org/~sunilg/hadoop-3.1.2-RC1/
> >>>
> >>> The RC tag in git is release-3.1.2-RC1:
> >>> https://github.com/apache/hadoop/commits/release-3.1.2-RC1
> >>>
> >>> The maven artifacts are available via repository.apache.org at
> >>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1215
> >>>
> >>> This vote will run 5 days from now.
> >>>
> >>> 3.1.2 contains 325 [1] fixed JIRA issues since 3.1.1.
> >>>
> >>> We have done testing with a pseudo cluster and distributed shell job.
> >>>
> >>> My +1 to start.
> >>>
> >>> Best,
> >>> Wangda Tan and Sunil Govindan
> >>>
> >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.1.2)
> >>> ORDER BY priority DESC
> >>>
> >>
>

[VOTE] Propose to start new Hadoop sub project "submarine"

[Wangda Tan]

Hi all,

According to positive feedbacks from the thread [1]

This is vote thread to start a new subproject named "hadoop-submarine"
which follows the release process already established for ozone.

The vote runs for usual 7 days, which ends at Feb 8th 5 PM PDT.

Thanks,
Wangda Tan

[1]
https://lists.apache.org/thread.html/f864461eb188bd12859d51b0098ec38942c4429aae7e4d001a633d96@%3Cyarn-dev.hadoop.apache.org%3E

Re: [DISCUSS] Making submarine to different release model like Ozone

[Wangda Tan]

Eric,
Thanks for your reconsideration. We will definitely try best to not break
compatibilities, etc. like how we did to other components!

Really appreciate everybody's support, thoughts, suggestions shared on this
thread. Given the discussion went very positive, I will go ahead to send a
voting thread.

Best,
Wangda

On Fri, Feb 1, 2019 at 2:06 PM Eric Yang  wrote:

> If HDFS or YARN breaks compatibility with Submarine, it will require to
> make release to catch up with the latest Hadoop changes.  On
> hadoop.apache.org website, the latest news may always have Submarine on
> top to repair compatibility with latest of Hadoop.  This may overwhelm any
> interesting news that may happen in Hadoop space.  I don’t like to see that
> happen, but unavoidable with independent release cycle.  Maybe there is a
> good way to avoid this with help of release manager to ensure that
> Hadoop/Submarine don’t break compatibility frequently.
>
>
>
> For me to lift my veto, release managers of independent release cycles
> need to take responsibility to ensure X version of Hadoop is tested with Y
> version of Submarine.  Release managers will have to do more work to ensure
> the defined combination works.  With the greater responsibility of release
> management comes with its own reward.  Seasoned PMC may be nominated to
> become Apache Member, which will help with Submarine to enter Apache
> Incubator when time is right.  Hence, I will withdraw my veto and let
> Submarine set its own course.
>
>
>
> Good luck Wangda.
>
>
>
> Regards,
>
> Eric
>
>
>
> *From: *Wangda Tan 
> *Date: *Friday, February 1, 2019 at 10:52 AM
> *To: *Eric Yang 
> *Cc: *Weiwei Yang , Xun Liu ,
> Hadoop Common , "yarn-dev@hadoop.apache.org"
> , Hdfs-dev , "
> mapreduce-...@hadoop.apache.org" 
> *Subject: *Re: [DISCUSS] Making submarine to different release model like
> Ozone
>
>
>
> Thanks everyone for sharing thoughts!
>
>
>
> Eric, appreciate your suggestions. But there are many examples to have
> separate releases, like Hive's storage API, OZone, etc. For loosely coupled
> sub-projects, it gonna be great (at least for most of the users) to have
> separate releases so new features can be faster consumed and iterated. From
> above feedbacks from developers and users, I think it is also what people
> want.
>
>
>
> Another concern you mentioned is Submarine is aligned with Hadoop project
> goals. From feedbacks we can see, it attracts companies continue using
> Hadoop to solve their ML/DL requirements, it also created a good feedback
> loop, many issues faced, and some new functionalities added by Submarine
> went back to Hadoop. Such as localization files, directories. GPU topology
> related enhancement, etc.
>
>
>
> We will definitely use this sub-project opportunity to fast grow both
> Submarine and Hadoop, try to get fast release cycles for both of the
> projects. And for your suggestion about Apache incubator, we can reconsider
> it once Submarine becomes a more independent project, now it is still too
> small and too much overhead to go through the process, I don't want to stop
> the fast-growing community for months to go through incubator process for
> now.
>
>
>
> I really hope my comment can help you reconsider the veto. :)
>
>
>
> Thanks,
>
> Wangda
>
>
>
> On Fri, Feb 1, 2019 at 9:39 AM Eric Yang  wrote:
>
> Submarine is an application built for YARN framework, but it does not have
> strong dependency on YARN development.  For this kind of projects, it would
> be best to enter Apache Incubator cycles to create a new community.  Apache
> commons is the only project other than Incubator that has independent
> release cycles.  The collection is large, and the project goal is
> ambitious.  No one really knows which component works with each other in
> Apache commons.  Hadoop is a much more focused project on distributed
> computing framework and not incubation sandbox.  For alignment with Hadoop
> goals, and we want to prevent Hadoop project to be overloaded while
> allowing good ideas to be carried forwarded in Apache incubator.  Put on my
> Apache Member hat, my vote is -1 to allow more independent subproject
> release cycle in Hadoop project that does not align with Hadoop project
> goals.
>
> Apache incubator process is highly recommended for Submarine:
> https://incubator.apache.org/policy/process.html This allows Submarine to
> develop for older version of Hadoop like Spark works with multiple versions
> of Hadoop.
>
> Regards,
> Eric
>
> On 1/31/19, 10:51 PM, "Weiwei Yang"  wrote:
>
> Thanks for proposing this Wangda, my +1 as well.
> It is amazing to see the

Re: [DISCUSS] Making submarine to different release model like Ozone

[Wangda Tan]

Thanks everyone for sharing thoughts!

Eric, appreciate your suggestions. But there are many examples to have
separate releases, like Hive's storage API, OZone, etc. For loosely coupled
sub-projects, it gonna be great (at least for most of the users) to have
separate releases so new features can be faster consumed and iterated. From
above feedbacks from developers and users, I think it is also what people
want.

Another concern you mentioned is Submarine is aligned with Hadoop project
goals. From feedbacks we can see, it attracts companies continue using
Hadoop to solve their ML/DL requirements, it also created a good feedback
loop, many issues faced, and some new functionalities added by Submarine
went back to Hadoop. Such as localization files, directories. GPU topology
related enhancement, etc.

We will definitely use this sub-project opportunity to fast grow both
Submarine and Hadoop, try to get fast release cycles for both of the
projects. And for your suggestion about Apache incubator, we can reconsider
it once Submarine becomes a more independent project, now it is still too
small and too much overhead to go through the process, I don't want to stop
the fast-growing community for months to go through incubator process for
now.

I really hope my comment can help you reconsider the veto. :)

Thanks,
Wangda

On Fri, Feb 1, 2019 at 9:39 AM Eric Yang  wrote:

> Submarine is an application built for YARN framework, but it does not have
> strong dependency on YARN development.  For this kind of projects, it would
> be best to enter Apache Incubator cycles to create a new community.  Apache
> commons is the only project other than Incubator that has independent
> release cycles.  The collection is large, and the project goal is
> ambitious.  No one really knows which component works with each other in
> Apache commons.  Hadoop is a much more focused project on distributed
> computing framework and not incubation sandbox.  For alignment with Hadoop
> goals, and we want to prevent Hadoop project to be overloaded while
> allowing good ideas to be carried forwarded in Apache incubator.  Put on my
> Apache Member hat, my vote is -1 to allow more independent subproject
> release cycle in Hadoop project that does not align with Hadoop project
> goals.
>
> Apache incubator process is highly recommended for Submarine:
> https://incubator.apache.org/policy/process.html This allows Submarine to
> develop for older version of Hadoop like Spark works with multiple versions
> of Hadoop.
>
> Regards,
> Eric
>
> On 1/31/19, 10:51 PM, "Weiwei Yang"  wrote:
>
> Thanks for proposing this Wangda, my +1 as well.
> It is amazing to see the progress made in Submarine last year, the
> community grows fast and quiet collaborative. I can see the reasons to get
> it release faster in its own cycle. And at the same time, the Ozone way
> works very well.
>
> —
> Weiwei
> On Feb 1, 2019, 10:49 AM +0800, Xun Liu , wrote:
> > +1
> >
> > Hello everyone,
> >
> > I am Xun Liu, the head of the machine learning team at Netease
> Research Institute. I quite agree with Wangda.
> >
> > Our team is very grateful for getting Submarine machine learning
> engine from the community.
> > We are heavy users of Submarine.
> > Because Submarine fits into the direction of our big data team's
> hadoop technology stack,
> > It avoids the needs to increase the manpower investment in learning
> other container scheduling systems.
> > The important thing is that we can use a common YARN cluster to run
> machine learning,
> > which makes the utilization of server resources more efficient, and
> reserves a lot of human and material resources in our previous years.
> >
> > Our team have finished the test and deployment of the Submarine and
> will provide the service to our e-commerce department (
> http://www.kaola.com/) shortly.
> >
> > We also plan to provides the Submarine engine in our existing YARN
> cluster in the next six months.
> > Because we have a lot of product departments need to use machine
> learning services,
> > for example:
> > 1) Game department (http://game.163.com/) needs AI battle training,
> > 2) News department (http://www.163.com) needs news recommendation,
> > 3) Mailbox department (http://www.163.com) requires anti-spam and
> illegal detection,
> > 4) Music department (https://music.163.com/) requires music
> recommendation,
> > 5) Education department (http://www.youdao.com) requires voice
> recognition,
> > 6) Massive Open Online Courses (https://open.163.com/) requires
> multilingual translation and so on.
> >
>

[DISCUSS] Making submarine to different release model like Ozone

[Wangda Tan]

Hi devs,

Since we started submarine-related effort last year, we received a lot of
feedbacks, several companies (such as Netease, China Mobile, etc.)  are
trying to deploy Submarine to their Hadoop cluster along with big data
workloads. Linkedin also has big interests to contribute a Submarine TonY (
https://github.com/linkedin/TonY) runtime to allow users to use the same
interface.

>From what I can see, there're several issues of putting Submarine under
yarn-applications directory and have same release cycle with Hadoop:

1) We started 3.2.0 release at Sep 2018, but the release is done at Jan
2019. Because of non-predictable blockers and security issues, it got
delayed a lot. We need to iterate submarine fast at this point.

2) We also see a lot of requirements to use Submarine on older Hadoop
releases such as 2.x. Many companies may not upgrade Hadoop to 3.x in a
short time, but the requirement to run deep learning is urgent to them. We
should decouple Submarine from Hadoop version.

And why we wanna to keep it within Hadoop? First, Submarine included some
innovation parts such as enhancements of user experiences for YARN
services/containerization support which we can add it back to Hadoop later
to address common requirements. In addition to that, we have a big overlap
in the community developing and using it.

There're several proposals we have went through during Ozone merge to trunk
discussion:
https://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201803.mbox/%3ccahfhakh6_m3yldf5a2kq8+w-5fbvx5ahfgs-x1vajw8gmnz...@mail.gmail.com%3E

I propose to adopt Ozone model: which is the same master branch, different
release cycle, and different release branch. It is a great example to show
agile release we can do (2 Ozone releases after Oct 2018) with less
overhead to setup CI, projects, etc.

*Links:*
- JIRA: https://issues.apache.org/jira/browse/YARN-8135
- Design doc
<https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit>
- User doc
<https://hadoop.apache.org/docs/r3.2.0/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/Index.html>
(3.2.0
release)
- Blogposts, {Submarine} : Running deep learning workloads on Apache Hadoop
<https://hortonworks.com/blog/submarine-running-deep-learning-workloads-apache-hadoop/>,
(Chinese Translation: Link <https://www.jishuwen.com/d/2Vpu>)
- Talks: Strata Data Conf NY
<https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/68289>

Thoughts?

Thanks,
Wangda Tan

Re: [VOTE] Release Apache Hadoop 3.1.2 - RC0

[Wangda Tan]

Hi Rohith,

Thanks for reporting and fixing this issue, let’s redo the issue and will
send another vote shortly.

Best,
Wangda

On Sun, Jan 27, 2019 at 10:03 PM Rohith Sharma K S <
rohithsharm...@apache.org> wrote:

> @Wangda Tan  I have pushed the changes to branch-3.1
> and branch-3.1.2. Verified hadoop-3.1.2 branch build for functionalities.
> Should new RC to be given OR same could be continued with updated
> repositories?
>
> On Mon, 28 Jan 2019 at 10:54, Rohith Sharma K S 
> wrote:
>
>> -1, I found an issue in ATSv2 initialization in NodeManager.  This causes
>> none of the ATSv2 events published from NodeManager. I have created
>> YARN-9242 for tracking the same.
>>
>> -Rohith Sharma K S
>>
>> On Fri, 25 Jan 2019 at 11:40, Wangda Tan  wrote:
>>
>>> Hi folks,
>>>
>>> With tons of help from Sunil, we have created RC0 for Apache Hadoop
>>> 3.1.2.
>>> The artifacts are available here:
>>>
>>> *http://home.apache.org/~sunilg/hadoop-3.1.2-RC0/
>>> <http://home.apache.org/~sunilg/hadoop-3.1.2-RC0/>*
>>>
>>> The RC tag in git is release-3.1.2-RC0:
>>> https://github.com/apache/hadoop/commits/release-3.1.2-RC0
>>>
>>> The maven artifacts are available via repository.apache.org at
>>> *
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1212/
>>> <
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1212/
>>> >*
>>>
>>> This vote will run 5 days from now.
>>>
>>> 3.1.2 contains 325 [1] fixed JIRA issues since 3.1.1.
>>>
>>> I have done testing with a pseudo cluster and distributed shell job. My
>>> +1
>>> to start.
>>>
>>> Best,
>>> Wangda Tan and Sunil Govind
>>>
>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.2)
>>> ORDER BY priority DESC
>>>
>> --
Sent from Gmail Mobile

[VOTE] Release Apache Hadoop 3.1.2 - RC0

[Wangda Tan]

Hi folks,

With tons of help from Sunil, we have created RC0 for Apache Hadoop 3.1.2.
The artifacts are available here:

*http://home.apache.org/~sunilg/hadoop-3.1.2-RC0/
<http://home.apache.org/~sunilg/hadoop-3.1.2-RC0/>*

The RC tag in git is release-3.1.2-RC0:
https://github.com/apache/hadoop/commits/release-3.1.2-RC0

The maven artifacts are available via repository.apache.org at
*https://repository.apache.org/content/repositories/orgapachehadoop-1212/
<https://repository.apache.org/content/repositories/orgapachehadoop-1212/>*

This vote will run 5 days from now.

3.1.2 contains 325 [1] fixed JIRA issues since 3.1.1.

I have done testing with a pseudo cluster and distributed shell job. My +1
to start.

Best,
Wangda Tan and Sunil Govind

[1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.2)
ORDER BY priority DESC

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

onfiguration due to changes in [orgapachehadoop-1202
> (staging: open)] made by wangda...
> 2019-01-22 05:40:50 INFO  [99598137-805254] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:40:50 INFO  [99598137-805254] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:40:50 INFO  [99598137-805254] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:40:50 INFO  [99598137-805254] -
> com.sonatype.nexus.staging.internal.task.RepositoryOpenTask - Staging
> repository created: orgapachehadoop-1202
> 2019-01-22 05:41:18 INFO  [99598137-805274] -
> com.sonatype.nexus.staging.internal.DefaultStagingManager - Closing staging
> repositories [orgapachehadoop-1202]
> 2019-01-22 05:41:18 INFO  [pool-1-thread-7] -
> com.sonatype.nexus.staging.internal.task.StagingBackgroundTask - STARTED
> Closing staging repositories: [orgapachehadoop-1202]
> 2019-01-22 05:41:18 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:18 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:18 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:20 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:20 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:20 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:20 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:20 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:20 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:20 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:20 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:20 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:20 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:22 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:22 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:22 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:22 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:22 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:22 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:22 WARN  [pool-1-thread-7] -
> com.sonatype.nexus.staging.internal.task.RepositoryCloseTask - One or more
> operations failed; aborting:
> com.sonatype.nexus.staging.StagingRulesFailedException: One or more rules
> have failed
> 2019-01-22 05:41:22 INFO  [pool-1-thread-7] -
> com.sonatype.nexus.staging.internal.task.StagingBackgroundTask - FINISHED
> Closing staging repositories: [orgapachehadoop-1202]
>
>
>
>
> On Jan 21, 2019, at 9:42 PM, Wangda Tan  wrote:
>
> Hi Chris,
>
> Thanks for helping the issue,
>
> Now the issue still exists but the process becomes much faster:
>
> failureMessage Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-project/3.1.2/hadoop-project-3.1.2.pom', check
> the logs.
> failureMessage Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-main/3.1.2/hadoop-main-3.1.2.pom', check the
> logs.
> failureMessage Failed to validate the pgp signature of
> '/org/apache/h

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

Hi Chris,

Thanks for helping the issue,

Now the issue still exists but the process becomes much faster:

failureMessage Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-project/3.1.2/hadoop-project-3.1.2.pom', check
the logs.
failureMessage Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-main/3.1.2/hadoop-main-3.1.2.pom', check the
logs.
failureMessage Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-build-tools/3.1.2/hadoop-build-tools-3.1.2.jar',
check the logs.
failureMessage Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-build-tools/3.1.2/hadoop-build-tools-3.1.2.pom',
check the logs.

Above is the error message, is it possible to get the actual failure
message from Nexus server?

Thanks,
Wangda

On Mon, Jan 21, 2019 at 5:18 PM Chris Lambertus  wrote:

> It looks like there are timeouts from some of the keyservers. I’ve trimmed
> the list again to only servers known to be working (ubuntu and
> sks-keyservers.net <http://sks-keyservers.net/>). Can you give it a try
> again?
>
> Brian, there are also a number of timeout errors related to central, but I
> think they are unrelated.
>
> com.sonatype.central.secure.nexus.plugin.internal.AuthtokenFetcherImpl -
> Failed to fetch authtoken: org.apache.http.conn.ConnectTimeoutException:
> Connect to secure.central.sonatype.com:443 [
> secure.central.sonatype.com/207.223.241.90] failed: connect timed out
>
> -Chris
>
>
> > On Jan 21, 2019, at 2:39 PM, Brian Fox  wrote:
> >
> > They keys file is irrelevant to Nexus. The only thing that matters is
> it’s in the mit pgp key ring.
> >
> > --Brian (mobile)
> >
> >
> > On Jan 21, 2019, at 3:34 PM, Wangda Tan  wheele...@gmail.com>> wrote:
> >
> >> I just checked on KEYS file, it doesn't show sig part. I updated KEYS
> file on Apache
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS <
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS> and made
> it be ultimately trusted.
> >>
> >> pub   rsa4096 2018-03-20 [SC]
> >>   4C899853CDDA4E40C60212B5B3FA653D57300D45
> >> uid   [ultimate] Wangda tan  wan...@apache.org>>
> >> sig 3B3FA653D57300D45 2018-03-20  Wangda tan  <mailto:wan...@apache.org>>
> >> sub   rsa4096 2018-03-20 [E]
> >> sig      B3FA653D57300D45 2018-03-20  Wangda tan  <mailto:wan...@apache.org>>
> >> But the error still remains same while closing repo, not sure how to
> get it resolved ..
> >>
> >>
> >> On Mon, Jan 21, 2019 at 9:14 AM Wangda Tan  <mailto:wheele...@gmail.com>> wrote:
> >> Hi David,
> >>
> >> Thanks for helping check this,
> >>
> >> I can see signatures on my key:
> >>
> >> pub  4096R/57300D45 <
> http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45> 2018-03-20
>
> >>   Fingerprint=4C89 9853 CDDA 4E40 C602  12B5 B3FA 653D 5730 0D45
> >>
> >> uid Wangda tan mailto:wan...@apache.org>>
> >> sig  sig3  57300D45 <
> http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45> 2018-03-20
> __ __ [selfsig] <
> http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0xB3FA653D57300D45
> >
> >> sig  sig   C36C5F0F <
> http://pool.sks-keyservers.net:11371/key/0x255ADF56C36C5F0F> 2018-04-05
> __ __ Vinod Kumar Vavilapalli (I am also known as
> @tshooter.)  <
> http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0x255ADF56C36C5F0F
> >
> >> sig  sig   F9CBBD4C <
> http://pool.sks-keyservers.net:11371/key/0x29ED86E6F9CBBD4C> 2018-11-08
> __ __ shikong  <
> http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0x29ED86E6F9CBBD4C
> >
> >>
> >> sub  4096R/D0C16F12 2018-03-20
> >> sig sbind  57300D45 <
> http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45> 2018-03-20
> __ __ [] <
> http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0xB3FA653D57300D45
> >
> >> And gpg --edit-key also shows:
> >>
> >> gpg --edit-key 4C899853CDDA4E40C60212B5B3FA653D57300D45
> >> gpg (GnuPG) 2.2.5; Copyright (C) 2018 Free Software Foundation, Inc.
> >> This is free software: you are free to change and redistribute it.
> >> There is NO WARRANTY, to the extent permitted by law.
> >>
> >> Secret key is available.
> >>
> >> sec  rsa4096/B3FA653D57300D45
> >>  created: 2018-03-20  expires: never   usage: SC
> >>  trust: unknown   validity: unkn

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

I just checked on KEYS file, it doesn't show sig part. I updated KEYS file
on Apache https://dist.apache.org/repos/dist/release/hadoop/common/KEYS and
made it be ultimately trusted.

pub   rsa4096 2018-03-20 [SC]
  4C899853CDDA4E40C60212B5B3FA653D57300D45
uid   [ultimate] Wangda tan 
sig 3B3FA653D57300D45 2018-03-20  Wangda tan 
sub   rsa4096 2018-03-20 [E]
sig  B3FA653D57300D45 2018-03-20  Wangda tan 

But the error still remains same while closing repo, not sure how to
get it resolved ..



On Mon, Jan 21, 2019 at 9:14 AM Wangda Tan  wrote:

> Hi David,
>
> Thanks for helping check this,
>
> I can see signatures on my key:
>
> pub  4096R/57300D45 
> <http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45> 2018-03-20
>Fingerprint=4C89 9853 CDDA 4E40 C602  12B5 B3FA 653D 5730 0D45
> uid Wangda tan 
> sig  sig3  57300D45 
> <http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45> 2018-03-20 
> __ __ [selfsig] 
> <http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0xB3FA653D57300D45>
> sig  sig   C36C5F0F 
> <http://pool.sks-keyservers.net:11371/key/0x255ADF56C36C5F0F> 2018-04-05 
> __ __ Vinod Kumar Vavilapalli (I am also known as @tshooter.) 
>  
> <http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0x255ADF56C36C5F0F>
> sig  sig   F9CBBD4C 
> <http://pool.sks-keyservers.net:11371/key/0x29ED86E6F9CBBD4C> 2018-11-08 
> __ __ shikong  
> <http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0x29ED86E6F9CBBD4C>
> sub  4096R/D0C16F12 2018-03-20
> sig sbind  57300D45 
> <http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45> 2018-03-20 
> __ __ [] 
> <http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0xB3FA653D57300D45>
>
> And gpg --edit-key also shows:
>
> gpg --edit-key 4C899853CDDA4E40C60212B5B3FA653D57300D45
> gpg (GnuPG) 2.2.5; Copyright (C) 2018 Free Software Foundation, Inc.
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
>
> Secret key is available.
>
> sec  rsa4096/B3FA653D57300D45
>  created: 2018-03-20  expires: never       usage: SC
>  trust: unknown   validity: unknown
> ssb  rsa4096/79CD893FD0C16F12
>  created: 2018-03-20  expires: never   usage: E
> [ unknown] (1). Wangda tan 
>
> Thanks,
> Wangda
>
> On Mon, Jan 21, 2019 at 9:08 AM David Nalley  wrote:
>
>> I wonder if it's because there are no signatures on your key.
>>
>> --David
>>
>> On Mon, Jan 21, 2019 at 11:57 AM Wangda Tan  wrote:
>> >
>> > Hi Brian,
>> >
>> > Here're links to my key:
>> >
>> > http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45
>> >
>> > http://pgp.mit.edu/pks/lookup?op=get=0xB3FA653D57300D45
>> >
>> > On Apache SVN:
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >
>> > Thanks,
>> > Wangda
>> >
>> > On Mon, Jan 21, 2019 at 6:51 AM Brian Demers 
>> wrote:
>> >>
>> >> Can you share the link to your key?
>> >>
>> >> -Brian
>> >>
>> >> On Jan 20, 2019, at 11:21 PM, Wangda Tan  wrote:
>> >>
>> >> Still couldn't figure out without locating the log on the Nexus
>> machine. With help from several committers and PMCs, we didn't see anything
>> wrong with my signing key.
>> >>
>> >> I don't want to delay 3.1.2 more because of this. Is it allowed for me
>> to publish artifacts (like tarball, source package, etc.) only and somebody
>> else to push Maven bits to Nexus. I believe Apache bylaw should allow that
>> because there're several releases have more than one release managers. If
>> it is not allowed, please take over the RM role if you have the bandwidth,
>> I think most works have been done except close the Nexus repo.
>> >>
>> >> Thanks,
>> >> Wangda
>> >>
>> >> On Thu, Jan 17, 2019 at 11:18 AM Wangda Tan 
>> wrote:
>> >>>
>> >>> Spent several more hours trying to figure out the issue, still no
>> luck.
>> >>>
>> >>> I just filed https://issues.sonatype.org/browse/OSSRH-45646, really
>> appreciate if anybody could add some suggestions.
>> >>>
>> >>> Thanks,
>> >>> Wangda
>> >>>
>> >>> On Tue, Jan 15, 2019 at 9:48 AM Wangda Tan 
>> wrote:
>> >>>>
>> >>>> It 

Re: [NOTICE] branch-3.1.2 created for 3.1.2 release

[Wangda Tan]

Hi devs,

Caused by delays in releasing preparing, we plan to roll RC1 tomorrow.
Given the key issue is not resolved yet, I asked Sunil to be co-RM for this
release.

branch-3.1.2 is open for critical/blocker changes only, please add your
fixes to the branch, we will start rolling RC tomorrow (Tue) 9 AM PDT.

Thank,
Wangda


On Mon, Jan 7, 2019 at 10:13 AM Wangda Tan  wrote:

> Hi devs,
>
> I just created branch-3.1.2 for 3.1.2 release and pushed to remote.
> branch-3.1 is still open for new commits required for next 3.1.x release
> (which will be 3.1.3). In the meantime, branch-3.1.2 is closed for new
> commits unless further noticed. I'm working on RC0 of 3.1.2 release now.
>
> Best,
> Wangda
>

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

Hi David,

Thanks for helping check this,

I can see signatures on my key:

pub  4096R/57300D45
<http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45>
2018-03-20
 Fingerprint=4C89 9853 CDDA 4E40 C602  12B5 B3FA 653D 5730 0D45
uid Wangda tan 
sig  sig3  57300D45
<http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45>
2018-03-20 __ __ [selfsig]
<http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0xB3FA653D57300D45>
sig  sig   C36C5F0F
<http://pool.sks-keyservers.net:11371/key/0x255ADF56C36C5F0F>
2018-04-05 __ __ Vinod Kumar Vavilapalli (I am also
known as @tshooter.) 
<http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0x255ADF56C36C5F0F>
sig  sig   F9CBBD4C
<http://pool.sks-keyservers.net:11371/key/0x29ED86E6F9CBBD4C>
2018-11-08 __ __ shikong 
<http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0x29ED86E6F9CBBD4C>
sub  4096R/D0C16F12 2018-03-20
sig sbind  57300D45
<http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45>
2018-03-20 __ __ []
<http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0xB3FA653D57300D45>

And gpg --edit-key also shows:

gpg --edit-key 4C899853CDDA4E40C60212B5B3FA653D57300D45
gpg (GnuPG) 2.2.5; Copyright (C) 2018 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Secret key is available.

sec  rsa4096/B3FA653D57300D45
 created: 2018-03-20  expires: never   usage: SC
 trust: unknown   validity: unknown
ssb  rsa4096/79CD893FD0C16F12
 created: 2018-03-20  expires: never   usage: E
[ unknown] (1). Wangda tan 

Thanks,
Wangda

On Mon, Jan 21, 2019 at 9:08 AM David Nalley  wrote:

> I wonder if it's because there are no signatures on your key.
>
> --David
>
> On Mon, Jan 21, 2019 at 11:57 AM Wangda Tan  wrote:
> >
> > Hi Brian,
> >
> > Here're links to my key:
> >
> > http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45
> >
> > http://pgp.mit.edu/pks/lookup?op=get=0xB3FA653D57300D45
> >
> > On Apache SVN:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Thanks,
> > Wangda
> >
> > On Mon, Jan 21, 2019 at 6:51 AM Brian Demers 
> wrote:
> >>
> >> Can you share the link to your key?
> >>
> >> -Brian
> >>
> >> On Jan 20, 2019, at 11:21 PM, Wangda Tan  wrote:
> >>
> >> Still couldn't figure out without locating the log on the Nexus
> machine. With help from several committers and PMCs, we didn't see anything
> wrong with my signing key.
> >>
> >> I don't want to delay 3.1.2 more because of this. Is it allowed for me
> to publish artifacts (like tarball, source package, etc.) only and somebody
> else to push Maven bits to Nexus. I believe Apache bylaw should allow that
> because there're several releases have more than one release managers. If
> it is not allowed, please take over the RM role if you have the bandwidth,
> I think most works have been done except close the Nexus repo.
> >>
> >> Thanks,
> >> Wangda
> >>
> >> On Thu, Jan 17, 2019 at 11:18 AM Wangda Tan 
> wrote:
> >>>
> >>> Spent several more hours trying to figure out the issue, still no luck.
> >>>
> >>> I just filed https://issues.sonatype.org/browse/OSSRH-45646, really
> appreciate if anybody could add some suggestions.
> >>>
> >>> Thanks,
> >>> Wangda
> >>>
> >>> On Tue, Jan 15, 2019 at 9:48 AM Wangda Tan 
> wrote:
> >>>>
> >>>> It seems the problem still exists for me:
> >>>>
> >>>> Now the error message only contains:
> >>>>
> >>>> failureMessage  Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-client-check-invariants/3.1.2/hadoop-client-check-invariants-3.1.2.pom',
> check the logs.
> >>>> failureMessage  Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-resourceestimator/3.1.2/hadoop-resourceestimator-3.1.2-javadoc.jar',
> check the logs.
> >>>>
> >>>> If anybody has access the Nexus node, could you please help to check
> what is the failure message?
> >>>>
> >>>> Thanks,
> >>>> Wangda
> >>>>
> >>>>
> >>>> On Tue, Jan 15, 2019 at 9:56 AM Brian Fox  wrote:
> >>>>>
> >>>>> Good to know. The pool has occasionally had sync issues, but we're
> talking 3 times in the last 8-9 years.
>

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

Hi Brian,

Here're links to my key:

http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45

http://pgp.mit.edu/pks/lookup?op=get=0xB3FA653D57300D45

On Apache SVN: https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Thanks,
Wangda

On Mon, Jan 21, 2019 at 6:51 AM Brian Demers  wrote:

> Can you share the link to your key?
>
> -Brian
>
> On Jan 20, 2019, at 11:21 PM, Wangda Tan  wrote:
>
> Still couldn't figure out without locating the log on the Nexus machine.
> With help from several committers and PMCs, we didn't see anything wrong
> with my signing key.
>
> I don't want to delay 3.1.2 more because of this. Is it allowed for me to
> publish artifacts (like tarball, source package, etc.) only and somebody
> else to push Maven bits to Nexus. I believe Apache bylaw should allow that
> because there're several releases have more than one release managers. If
> it is not allowed, please take over the RM role if you have the bandwidth,
> I think most works have been done except close the Nexus repo.
>
> Thanks,
> Wangda
>
> On Thu, Jan 17, 2019 at 11:18 AM Wangda Tan  wrote:
>
>> Spent several more hours trying to figure out the issue, still no luck.
>>
>> I just filed https://issues.sonatype.org/browse/OSSRH-45646, really
>> appreciate if anybody could add some suggestions.
>>
>> Thanks,
>> Wangda
>>
>> On Tue, Jan 15, 2019 at 9:48 AM Wangda Tan  wrote:
>>
>>> It seems the problem still exists for me:
>>>
>>> Now the error message only contains:
>>>
>>> failureMessage  Failed to validate the pgp signature of
>>> '/org/apache/hadoop/hadoop-client-check-invariants/3.1.2/hadoop-client-check-invariants-3.1.2.pom',
>>> check the logs.
>>> failureMessage  Failed to validate the pgp signature of
>>> '/org/apache/hadoop/hadoop-resourceestimator/3.1.2/hadoop-resourceestimator-3.1.2-javadoc.jar',
>>> check the logs.
>>>
>>> If anybody has access the Nexus node, could you please help to check
>>> what is the failure message?
>>>
>>> Thanks,
>>> Wangda
>>>
>>>
>>> On Tue, Jan 15, 2019 at 9:56 AM Brian Fox  wrote:
>>>
>>>> Good to know. The pool has occasionally had sync issues, but we're
>>>> talking 3 times in the last 8-9 years.
>>>>
>>>> On Tue, Jan 15, 2019 at 10:39 AM Elek, Marton  wrote:
>>>>
>>>>> My key was pushed to the server with pgp about 1 year ago, and it
>>>>> worked
>>>>> well with the last Ratis release. So it should be synced between the
>>>>> key
>>>>> servers.
>>>>>
>>>>> But it seems that the INFRA solved the problem with shuffling the key
>>>>> server order (or it was an intermittent issue): see INFRA-17649
>>>>>
>>>>> Seems to be working now...
>>>>>
>>>>> Marton
>>>>>
>>>>>
>>>>> On 1/15/19 5:19 AM, Wangda Tan wrote:
>>>>> > HI Brain,
>>>>> > Thanks for responding, could u share how to push to keys to Apache
>>>>> pgp pool?
>>>>> >
>>>>> > Best,
>>>>> > Wangda
>>>>> >
>>>>> > On Mon, Jan 14, 2019 at 10:44 AM Brian Fox 
>>>>> wrote:
>>>>> >
>>>>> >> Did you push your key up to the pgp pool? That's what Nexus is
>>>>> validating
>>>>> >> against. It might take time to propagate if you just pushed it.
>>>>> >>
>>>>> >> On Mon, Jan 14, 2019 at 9:59 AM Elek, Marton 
>>>>> wrote:
>>>>> >>
>>>>> >>> Seems to be an INFRA issue for me:
>>>>> >>>
>>>>> >>> 1. I downloaded a sample jar file [1] + the signature from the
>>>>> >>> repository and it was ok, locally I verified it.
>>>>> >>>
>>>>> >>> 2. I tested it with an other Apache project (Ratis) and my key. I
>>>>> got
>>>>> >>> the same problem even if it worked at last year during the 0.3.0
>>>>> >>> release. (I used exactly the same command)
>>>>> >>>
>>>>> >>> I opened an infra ticket to check the logs of the Nexus as it was
>>>>> >>> suggested in the error message:
>>>>> >>>
>>>>> >>> https://issues.apache.org/jira/browse/I

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

Still couldn't figure out without locating the log on the Nexus machine.
With help from several committers and PMCs, we didn't see anything wrong
with my signing key.

I don't want to delay 3.1.2 more because of this. Is it allowed for me to
publish artifacts (like tarball, source package, etc.) only and somebody
else to push Maven bits to Nexus. I believe Apache bylaw should allow that
because there're several releases have more than one release managers. If
it is not allowed, please take over the RM role if you have the bandwidth,
I think most works have been done except close the Nexus repo.

Thanks,
Wangda

On Thu, Jan 17, 2019 at 11:18 AM Wangda Tan  wrote:

> Spent several more hours trying to figure out the issue, still no luck.
>
> I just filed https://issues.sonatype.org/browse/OSSRH-45646, really
> appreciate if anybody could add some suggestions.
>
> Thanks,
> Wangda
>
> On Tue, Jan 15, 2019 at 9:48 AM Wangda Tan  wrote:
>
>> It seems the problem still exists for me:
>>
>> Now the error message only contains:
>>
>> failureMessage  Failed to validate the pgp signature of
>> '/org/apache/hadoop/hadoop-client-check-invariants/3.1.2/hadoop-client-check-invariants-3.1.2.pom',
>> check the logs.
>> failureMessage  Failed to validate the pgp signature of
>> '/org/apache/hadoop/hadoop-resourceestimator/3.1.2/hadoop-resourceestimator-3.1.2-javadoc.jar',
>> check the logs.
>>
>> If anybody has access the Nexus node, could you please help to check what
>> is the failure message?
>>
>> Thanks,
>> Wangda
>>
>>
>> On Tue, Jan 15, 2019 at 9:56 AM Brian Fox  wrote:
>>
>>> Good to know. The pool has occasionally had sync issues, but we're
>>> talking 3 times in the last 8-9 years.
>>>
>>> On Tue, Jan 15, 2019 at 10:39 AM Elek, Marton  wrote:
>>>
>>>> My key was pushed to the server with pgp about 1 year ago, and it worked
>>>> well with the last Ratis release. So it should be synced between the key
>>>> servers.
>>>>
>>>> But it seems that the INFRA solved the problem with shuffling the key
>>>> server order (or it was an intermittent issue): see INFRA-17649
>>>>
>>>> Seems to be working now...
>>>>
>>>> Marton
>>>>
>>>>
>>>> On 1/15/19 5:19 AM, Wangda Tan wrote:
>>>> > HI Brain,
>>>> > Thanks for responding, could u share how to push to keys to Apache
>>>> pgp pool?
>>>> >
>>>> > Best,
>>>> > Wangda
>>>> >
>>>> > On Mon, Jan 14, 2019 at 10:44 AM Brian Fox 
>>>> wrote:
>>>> >
>>>> >> Did you push your key up to the pgp pool? That's what Nexus is
>>>> validating
>>>> >> against. It might take time to propagate if you just pushed it.
>>>> >>
>>>> >> On Mon, Jan 14, 2019 at 9:59 AM Elek, Marton 
>>>> wrote:
>>>> >>
>>>> >>> Seems to be an INFRA issue for me:
>>>> >>>
>>>> >>> 1. I downloaded a sample jar file [1] + the signature from the
>>>> >>> repository and it was ok, locally I verified it.
>>>> >>>
>>>> >>> 2. I tested it with an other Apache project (Ratis) and my key. I
>>>> got
>>>> >>> the same problem even if it worked at last year during the 0.3.0
>>>> >>> release. (I used exactly the same command)
>>>> >>>
>>>> >>> I opened an infra ticket to check the logs of the Nexus as it was
>>>> >>> suggested in the error message:
>>>> >>>
>>>> >>> https://issues.apache.org/jira/browse/INFRA-17649
>>>> >>>
>>>> >>> Marton
>>>> >>>
>>>> >>>
>>>> >>> [1]:
>>>> >>>
>>>> >>>
>>>> https://repository.apache.org/service/local/repositories/orgapachehadoop-1183/content/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-javadoc.jar
>>>> >>>
>>>> >>>
>>>> >>> On 1/13/19 6:27 AM, Wangda Tan wrote:
>>>> >>>> Uploaded sample file and signature.
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> On Sat, Jan 12, 2019 at 9:18 PM Wangda Tan >>> >>>> &l

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

Spent several more hours trying to figure out the issue, still no luck.

I just filed https://issues.sonatype.org/browse/OSSRH-45646, really
appreciate if anybody could add some suggestions.

Thanks,
Wangda

On Tue, Jan 15, 2019 at 9:48 AM Wangda Tan  wrote:

> It seems the problem still exists for me:
>
> Now the error message only contains:
>
> failureMessage  Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-client-check-invariants/3.1.2/hadoop-client-check-invariants-3.1.2.pom',
> check the logs.
> failureMessage  Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-resourceestimator/3.1.2/hadoop-resourceestimator-3.1.2-javadoc.jar',
> check the logs.
>
> If anybody has access the Nexus node, could you please help to check what
> is the failure message?
>
> Thanks,
> Wangda
>
>
> On Tue, Jan 15, 2019 at 9:56 AM Brian Fox  wrote:
>
>> Good to know. The pool has occasionally had sync issues, but we're
>> talking 3 times in the last 8-9 years.
>>
>> On Tue, Jan 15, 2019 at 10:39 AM Elek, Marton  wrote:
>>
>>> My key was pushed to the server with pgp about 1 year ago, and it worked
>>> well with the last Ratis release. So it should be synced between the key
>>> servers.
>>>
>>> But it seems that the INFRA solved the problem with shuffling the key
>>> server order (or it was an intermittent issue): see INFRA-17649
>>>
>>> Seems to be working now...
>>>
>>> Marton
>>>
>>>
>>> On 1/15/19 5:19 AM, Wangda Tan wrote:
>>> > HI Brain,
>>> > Thanks for responding, could u share how to push to keys to Apache pgp
>>> pool?
>>> >
>>> > Best,
>>> > Wangda
>>> >
>>> > On Mon, Jan 14, 2019 at 10:44 AM Brian Fox  wrote:
>>> >
>>> >> Did you push your key up to the pgp pool? That's what Nexus is
>>> validating
>>> >> against. It might take time to propagate if you just pushed it.
>>> >>
>>> >> On Mon, Jan 14, 2019 at 9:59 AM Elek, Marton  wrote:
>>> >>
>>> >>> Seems to be an INFRA issue for me:
>>> >>>
>>> >>> 1. I downloaded a sample jar file [1] + the signature from the
>>> >>> repository and it was ok, locally I verified it.
>>> >>>
>>> >>> 2. I tested it with an other Apache project (Ratis) and my key. I got
>>> >>> the same problem even if it worked at last year during the 0.3.0
>>> >>> release. (I used exactly the same command)
>>> >>>
>>> >>> I opened an infra ticket to check the logs of the Nexus as it was
>>> >>> suggested in the error message:
>>> >>>
>>> >>> https://issues.apache.org/jira/browse/INFRA-17649
>>> >>>
>>> >>> Marton
>>> >>>
>>> >>>
>>> >>> [1]:
>>> >>>
>>> >>>
>>> https://repository.apache.org/service/local/repositories/orgapachehadoop-1183/content/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-javadoc.jar
>>> >>>
>>> >>>
>>> >>> On 1/13/19 6:27 AM, Wangda Tan wrote:
>>> >>>> Uploaded sample file and signature.
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> On Sat, Jan 12, 2019 at 9:18 PM Wangda Tan >> >>>> <mailto:wheele...@gmail.com>> wrote:
>>> >>>>
>>> >>>> Actually, among the hundreds of failed messages, the "No public
>>> key"
>>> >>>> issues still occurred several times:
>>> >>>>
>>> >>>> failureMessage  No public key: Key with id:
>>> (b3fa653d57300d45)
>>> >>>> was not able to be located on http://gpg-keyserver.de/.
>>> Upload
>>> >>>> your public key and try the operation again.
>>> >>>> failureMessage  No public key: Key with id:
>>> (b3fa653d57300d45)
>>> >>>> was not able to be located on
>>> >>>> http://pool.sks-keyservers.net:11371. Upload your public
>>> key
>>> >>> and
>>> >>>> try the operation again.
>>> >>>> failureMessage  No public key: Key with id:
>>> (b3fa653d57300d4

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC1

[Wangda Tan]

Manually checked the new repo, looks good, thanks Sunil!  +1

Best,
Wangda

On Tue, Jan 15, 2019 at 2:04 PM Sunil G  wrote:

> Thanks folks for voting.
>
> For the point mentioned from Zoltan, I re-ran the same source and deployed
> to Nexus to avail those missing source.
> https://repository.apache.org/content/repositories/orgapachehadoop-1186/
>
> Please help to cross check the same.
>
> Thanks & Regards
> Sunil
>
> On Tue, Jan 15, 2019 at 10:05 AM Wangda Tan  wrote:
>
>> +1 (Binding).
>>
>> Deployed a local cluster from binary, and ran some sample sanity jobs.
>>
>> Thanks Sunil for driving the release.
>>
>> Best,
>> Wangda
>>
>>
>> On Mon, Jan 14, 2019 at 11:26 AM Virajith Jalaparti 
>> wrote:
>>
>>> Thanks Sunil and others who have worked on the making this release
>>> happen!
>>>
>>> +1 (non-binding)
>>>
>>> - Built from source
>>> - Deployed a pseudo-distributed one node cluster
>>> - Ran basic wordcount, sort, pi jobs
>>> - Basic HDFS/WebHDFS commands
>>> - Ran all the ABFS driver tests against an ADLS Gen 2 account in EAST US
>>>
>>> Non-blockers (AFAICT): The following tests in ABFS (HADOOP-15407) fail:
>>> - For ACLs ({{ITestAzureBlobFilesystemAcl}}) -- However, I believe these
>>> have been fixed in trunk.
>>> - {{
>>> ITestAzureBlobFileSystemE2EScale#testWriteHeavyBytesToFileAcrossThreads}}
>>> fails with an OutOfMemoryError exception. I see the same failure on
>>> trunk as well.
>>>
>>>
>>> On Mon, Jan 14, 2019 at 6:21 AM Elek, Marton  wrote:
>>>
>>>> Thanks Sunil to manage this release.
>>>>
>>>> +1 (non-binding)
>>>>
>>>> 1. built from the source (with clean local maven repo)
>>>> 2. verified signatures + checksum
>>>> 3. deployed 3 node cluster to Google Kubernetes Engine with generated
>>>> k8s resources [1]
>>>> 4. Executed basic HDFS commands
>>>> 5. Executed basic yarn example jobs
>>>>
>>>> Marton
>>>>
>>>> [1]: FTR: resources:
>>>> https://github.com/flokkr/k8s/tree/master/examples/hadoop , generator:
>>>> https://github.com/elek/flekszible
>>>>
>>>>
>>>> On 1/8/19 12:42 PM, Sunil G wrote:
>>>> > Hi folks,
>>>> >
>>>> >
>>>> > Thanks to all of you who helped in this release [1] and for helping
>>>> to vote
>>>> > for RC0. I have created second release candidate (RC1) for Apache
>>>> Hadoop
>>>> > 3.2.0.
>>>> >
>>>> >
>>>> > Artifacts for this RC are available here:
>>>> >
>>>> > http://home.apache.org/~sunilg/hadoop-3.2.0-RC1/
>>>> >
>>>> >
>>>> > RC tag in git is release-3.2.0-RC1.
>>>> >
>>>> >
>>>> >
>>>> > The maven artifacts are available via repository.apache.org at
>>>> >
>>>> https://repository.apache.org/content/repositories/orgapachehadoop-1178/
>>>> >
>>>> >
>>>> > This vote will run 7 days (5 weekdays), ending on 14th Jan at 11:59
>>>> pm PST.
>>>> >
>>>> >
>>>> >
>>>> > 3.2.0 contains 1092 [2] fixed JIRA issues since 3.1.0. Below feature
>>>> > additions
>>>> >
>>>> > are the highlights of this release.
>>>> >
>>>> > 1. Node Attributes Support in YARN
>>>> >
>>>> > 2. Hadoop Submarine project for running Deep Learning workloads on
>>>> YARN
>>>> >
>>>> > 3. Support service upgrade via YARN Service API and CLI
>>>> >
>>>> > 4. HDFS Storage Policy Satisfier
>>>> >
>>>> > 5. Support Windows Azure Storage - Blob file system in Hadoop
>>>> >
>>>> > 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
>>>> >
>>>> > 7. Improvements in Router-based HDFS federation
>>>> >
>>>> >
>>>> >
>>>> > Thanks to Wangda, Vinod, Marton for helping me in preparing the
>>>> release.
>>>> >
>>>> > I have done few testing with my pseudo cluster. My +1 to start.
>>>> >
>>>> >
>>>> >
>>>> > Regards,
>>>> >
>>>> > Sunil
>>>> >
>>>> >
>>>> >
>>>> > [1]
>>>> >
>>>> >
>>>> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
>>>> >
>>>> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
>>>> (3.2.0)
>>>> > AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status =
>>>> Resolved
>>>> > ORDER BY fixVersion ASC
>>>> >
>>>>
>>>> -
>>>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>>>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>>>>
>>>>

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

It seems the problem still exists for me:

Now the error message only contains:

failureMessage  Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-client-check-invariants/3.1.2/hadoop-client-check-invariants-3.1.2.pom',
check the logs.
failureMessage  Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-resourceestimator/3.1.2/hadoop-resourceestimator-3.1.2-javadoc.jar',
check the logs.

If anybody has access the Nexus node, could you please help to check what
is the failure message?

Thanks,
Wangda


On Tue, Jan 15, 2019 at 9:56 AM Brian Fox  wrote:

> Good to know. The pool has occasionally had sync issues, but we're talking
> 3 times in the last 8-9 years.
>
> On Tue, Jan 15, 2019 at 10:39 AM Elek, Marton  wrote:
>
>> My key was pushed to the server with pgp about 1 year ago, and it worked
>> well with the last Ratis release. So it should be synced between the key
>> servers.
>>
>> But it seems that the INFRA solved the problem with shuffling the key
>> server order (or it was an intermittent issue): see INFRA-17649
>>
>> Seems to be working now...
>>
>> Marton
>>
>>
>> On 1/15/19 5:19 AM, Wangda Tan wrote:
>> > HI Brain,
>> > Thanks for responding, could u share how to push to keys to Apache pgp
>> pool?
>> >
>> > Best,
>> > Wangda
>> >
>> > On Mon, Jan 14, 2019 at 10:44 AM Brian Fox  wrote:
>> >
>> >> Did you push your key up to the pgp pool? That's what Nexus is
>> validating
>> >> against. It might take time to propagate if you just pushed it.
>> >>
>> >> On Mon, Jan 14, 2019 at 9:59 AM Elek, Marton  wrote:
>> >>
>> >>> Seems to be an INFRA issue for me:
>> >>>
>> >>> 1. I downloaded a sample jar file [1] + the signature from the
>> >>> repository and it was ok, locally I verified it.
>> >>>
>> >>> 2. I tested it with an other Apache project (Ratis) and my key. I got
>> >>> the same problem even if it worked at last year during the 0.3.0
>> >>> release. (I used exactly the same command)
>> >>>
>> >>> I opened an infra ticket to check the logs of the Nexus as it was
>> >>> suggested in the error message:
>> >>>
>> >>> https://issues.apache.org/jira/browse/INFRA-17649
>> >>>
>> >>> Marton
>> >>>
>> >>>
>> >>> [1]:
>> >>>
>> >>>
>> https://repository.apache.org/service/local/repositories/orgapachehadoop-1183/content/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-javadoc.jar
>> >>>
>> >>>
>> >>> On 1/13/19 6:27 AM, Wangda Tan wrote:
>> >>>> Uploaded sample file and signature.
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Sat, Jan 12, 2019 at 9:18 PM Wangda Tan > >>>> <mailto:wheele...@gmail.com>> wrote:
>> >>>>
>> >>>> Actually, among the hundreds of failed messages, the "No public
>> key"
>> >>>> issues still occurred several times:
>> >>>>
>> >>>> failureMessage  No public key: Key with id:
>> (b3fa653d57300d45)
>> >>>>     was not able to be located on http://gpg-keyserver.de/.
>> Upload
>> >>>> your public key and try the operation again.
>> >>>> failureMessage  No public key: Key with id:
>> (b3fa653d57300d45)
>> >>>> was not able to be located on
>> >>>> http://pool.sks-keyservers.net:11371. Upload your public key
>> >>> and
>> >>>> try the operation again.
>> >>>> failureMessage  No public key: Key with id:
>> (b3fa653d57300d45)
>> >>>> was not able to be located on http://pgp.mit.edu:11371.
>> Upload
>> >>>> your public key and try the operation again.
>> >>>>
>> >>>> Once the close operation returned, I will upload sample files
>> which
>> >>>> may help troubleshoot the issue.
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> On Sat, Jan 12, 2019 at 9:04 PM Wangda Tan > >>>> <mailto:wheele...@gmail.com>> wrote:
>> >>>>
>> >>>> Than

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC1

[Wangda Tan]

+1 (Binding).

Deployed a local cluster from binary, and ran some sample sanity jobs.

Thanks Sunil for driving the release.

Best,
Wangda


On Mon, Jan 14, 2019 at 11:26 AM Virajith Jalaparti 
wrote:

> Thanks Sunil and others who have worked on the making this release happen!
>
> +1 (non-binding)
>
> - Built from source
> - Deployed a pseudo-distributed one node cluster
> - Ran basic wordcount, sort, pi jobs
> - Basic HDFS/WebHDFS commands
> - Ran all the ABFS driver tests against an ADLS Gen 2 account in EAST US
>
> Non-blockers (AFAICT): The following tests in ABFS (HADOOP-15407) fail:
> - For ACLs ({{ITestAzureBlobFilesystemAcl}}) -- However, I believe these
> have been fixed in trunk.
> - {{
> ITestAzureBlobFileSystemE2EScale#testWriteHeavyBytesToFileAcrossThreads}}
> fails with an OutOfMemoryError exception. I see the same failure on trunk
> as well.
>
>
> On Mon, Jan 14, 2019 at 6:21 AM Elek, Marton  wrote:
>
>> Thanks Sunil to manage this release.
>>
>> +1 (non-binding)
>>
>> 1. built from the source (with clean local maven repo)
>> 2. verified signatures + checksum
>> 3. deployed 3 node cluster to Google Kubernetes Engine with generated
>> k8s resources [1]
>> 4. Executed basic HDFS commands
>> 5. Executed basic yarn example jobs
>>
>> Marton
>>
>> [1]: FTR: resources:
>> https://github.com/flokkr/k8s/tree/master/examples/hadoop , generator:
>> https://github.com/elek/flekszible
>>
>>
>> On 1/8/19 12:42 PM, Sunil G wrote:
>> > Hi folks,
>> >
>> >
>> > Thanks to all of you who helped in this release [1] and for helping to
>> vote
>> > for RC0. I have created second release candidate (RC1) for Apache Hadoop
>> > 3.2.0.
>> >
>> >
>> > Artifacts for this RC are available here:
>> >
>> > http://home.apache.org/~sunilg/hadoop-3.2.0-RC1/
>> >
>> >
>> > RC tag in git is release-3.2.0-RC1.
>> >
>> >
>> >
>> > The maven artifacts are available via repository.apache.org at
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1178/
>> >
>> >
>> > This vote will run 7 days (5 weekdays), ending on 14th Jan at 11:59 pm
>> PST.
>> >
>> >
>> >
>> > 3.2.0 contains 1092 [2] fixed JIRA issues since 3.1.0. Below feature
>> > additions
>> >
>> > are the highlights of this release.
>> >
>> > 1. Node Attributes Support in YARN
>> >
>> > 2. Hadoop Submarine project for running Deep Learning workloads on YARN
>> >
>> > 3. Support service upgrade via YARN Service API and CLI
>> >
>> > 4. HDFS Storage Policy Satisfier
>> >
>> > 5. Support Windows Azure Storage - Blob file system in Hadoop
>> >
>> > 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
>> >
>> > 7. Improvements in Router-based HDFS federation
>> >
>> >
>> >
>> > Thanks to Wangda, Vinod, Marton for helping me in preparing the release.
>> >
>> > I have done few testing with my pseudo cluster. My +1 to start.
>> >
>> >
>> >
>> > Regards,
>> >
>> > Sunil
>> >
>> >
>> >
>> > [1]
>> >
>> >
>> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
>> >
>> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
>> > AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
>> > ORDER BY fixVersion ASC
>> >
>>
>> -
>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>>
>>

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

HI Brain,
Thanks for responding, could u share how to push to keys to Apache pgp pool?

Best,
Wangda

On Mon, Jan 14, 2019 at 10:44 AM Brian Fox  wrote:

> Did you push your key up to the pgp pool? That's what Nexus is validating
> against. It might take time to propagate if you just pushed it.
>
> On Mon, Jan 14, 2019 at 9:59 AM Elek, Marton  wrote:
>
>> Seems to be an INFRA issue for me:
>>
>> 1. I downloaded a sample jar file [1] + the signature from the
>> repository and it was ok, locally I verified it.
>>
>> 2. I tested it with an other Apache project (Ratis) and my key. I got
>> the same problem even if it worked at last year during the 0.3.0
>> release. (I used exactly the same command)
>>
>> I opened an infra ticket to check the logs of the Nexus as it was
>> suggested in the error message:
>>
>> https://issues.apache.org/jira/browse/INFRA-17649
>>
>> Marton
>>
>>
>> [1]:
>>
>> https://repository.apache.org/service/local/repositories/orgapachehadoop-1183/content/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-javadoc.jar
>>
>>
>> On 1/13/19 6:27 AM, Wangda Tan wrote:
>> > Uploaded sample file and signature.
>> >
>> >
>> >
>> > On Sat, Jan 12, 2019 at 9:18 PM Wangda Tan > > <mailto:wheele...@gmail.com>> wrote:
>> >
>> > Actually, among the hundreds of failed messages, the "No public key"
>> > issues still occurred several times:
>> >
>> > failureMessage  No public key: Key with id: (b3fa653d57300d45)
>> > was not able to be located on http://gpg-keyserver.de/. Upload
>> > your public key and try the operation again.
>> > failureMessage  No public key: Key with id: (b3fa653d57300d45)
>> > was not able to be located on
>> > http://pool.sks-keyservers.net:11371. Upload your public key
>> and
>> > try the operation again.
>> > failureMessage  No public key: Key with id: (b3fa653d57300d45)
>> > was not able to be located on http://pgp.mit.edu:11371. Upload
>> > your public key and try the operation again.
>> >
>> > Once the close operation returned, I will upload sample files which
>> > may help troubleshoot the issue.
>> >
>> > Thanks,
>> >
>> > On Sat, Jan 12, 2019 at 9:04 PM Wangda Tan > > <mailto:wheele...@gmail.com>> wrote:
>> >
>> > Thanks David for the quick response!
>> >
>> > I just retried, now the "No public key" issue is gone. However,
>> > the issue:
>> >
>> > failureMessage  Failed to validate the pgp signature of
>> >
>>  
>> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-tests.jar',
>> > check the logs.
>> > failureMessage  Failed to validate the pgp signature of
>> >
>>  
>> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-test-sources.jar',
>> > check the logs.
>> > failureMessage  Failed to validate the pgp signature of
>> >
>>  
>> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2.pom',
>> > check the logs.
>> >
>> >
>> > Still exists and repeated hundreds of times. Do you know how to
>> > access the logs mentioned by above log?
>> >
>> > Best,
>> > Wangda
>> >
>> > On Sat, Jan 12, 2019 at 8:37 PM David Nalley > > <mailto:da...@gnsa.us>> wrote:
>> >
>> > On Sat, Jan 12, 2019 at 9:09 PM Wangda Tan
>> > mailto:wheele...@gmail.com>> wrote:
>> > >
>> > > Hi Devs,
>> > >
>> > > I'm currently rolling Hadoop 3.1.2 release candidate,
>> > however, I saw an issue when I try to close repo in Nexus.
>> > >
>> > > Logs of
>> https://repository.apache.org/#stagingRepositories
>> > (orgapachehadoop-1183) shows hundreds of lines of the
>> > following error:
>> > >
>> > > failureMessage  No public key: Key with id:
>> > (b

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

Uploaded sample file and signature.



On Sat, Jan 12, 2019 at 9:18 PM Wangda Tan  wrote:

> Actually, among the hundreds of failed messages, the "No public key"
> issues still occurred several times:
>
> failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
> able to be located on http://gpg-keyserver.de/. Upload your public key
> and try the operation again.
> failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
> able to be located on http://pool.sks-keyservers.net:11371. Upload your
> public key and try the operation again.
> failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
> able to be located on http://pgp.mit.edu:11371. Upload your public key
> and try the operation again.
>
> Once the close operation returned, I will upload sample files which may
> help troubleshoot the issue.
>
> Thanks,
>
> On Sat, Jan 12, 2019 at 9:04 PM Wangda Tan  wrote:
>
>> Thanks David for the quick response!
>>
>> I just retried, now the "No public key" issue is gone. However,  the
>> issue:
>>
>> failureMessage  Failed to validate the pgp signature of
>> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-tests.jar',
>> check the logs.
>> failureMessage  Failed to validate the pgp signature of
>> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-test-sources.jar',
>> check the logs.
>> failureMessage  Failed to validate the pgp signature of
>> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2.pom',
>> check the logs.
>>
>>
>> Still exists and repeated hundreds of times. Do you know how to access
>> the logs mentioned by above log?
>>
>> Best,
>> Wangda
>>
>> On Sat, Jan 12, 2019 at 8:37 PM David Nalley  wrote:
>>
>>> On Sat, Jan 12, 2019 at 9:09 PM Wangda Tan  wrote:
>>> >
>>> > Hi Devs,
>>> >
>>> > I'm currently rolling Hadoop 3.1.2 release candidate, however, I saw
>>> an issue when I try to close repo in Nexus.
>>> >
>>> > Logs of https://repository.apache.org/#stagingRepositories
>>> (orgapachehadoop-1183) shows hundreds of lines of the following error:
>>> >
>>> > failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
>>> able to be located on http://gpg-keyserver.de/. Upload your public key
>>> and try the operation again.
>>> > failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
>>> able to be located on http://pool.sks-keyservers.net:11371. Upload your
>>> public key and try the operation again.
>>> > failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
>>> able to be located on http://pgp.mit.edu:11371. Upload your public key
>>> and try the operation again.
>>> > ...
>>> > failureMessage  Failed to validate the pgp signature of
>>> '/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-tests.jar',
>>> check the logs.
>>> > failureMessage  Failed to validate the pgp signature of
>>> '/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-test-sources.jar',
>>> check the logs.
>>> > failureMessage  Failed to validate the pgp signature of
>>> '/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-sources.jar',
>>> check the logs.
>>> >
>>> >
>>> > This is the same key I used before (and finished two releases), the
>>> same environment I used before.
>>> >
>>> > I have tried more than 10 times in the last two days, no luck. And
>>> closing the repo takes almost one hour (Regular time is less than 1 min)
>>> and always fail at the last.
>>> >
>>> > I used following commands to validate key exists on key servers
>>> >
>>> > gpg --keyserver pgp.mit.edu --recv-keys 57300D45
>>> > gpg: WARNING: unsafe permissions on homedir '/Users/wtan/.gnupg'
>>> > gpg: key B3FA653D57300D45: 1 signature not checked due to a missing key
>>> > gpg: key B3FA653D57300D45: "Wangda tan " not
>>> changed
>>> > gpg: Total number processed: 1
>>> > gpg:  unchanged: 1
>>> >
>>> > gpg --keyserver pool.sks-keyservers.net --recv-keys B3FA653D57300D45
>>> > gpg: WARNING: unsafe permissions on homedir '/Users/wtan/.gnupg'
>>> > gpg: key B3FA653D57300D45: 1 signature not checked due to a missing key
>>> > gpg: key B3FA653D57300D45: "Wangda tan " not
>>> changed
>>> > gpg: Total number processed: 1
>>> > gpg:  unchanged: 1
>>> >
>>>
>>> Both of these report that your key was not found.
>>> I took the key from the KEYS file and uploaded it to both of those
>>> servers.
>>>
>>> You might try the release again and see if this resolves the issue.
>>>
>>

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

Actually, among the hundreds of failed messages, the "No public key" issues
still occurred several times:

failureMessage  No public key: Key with id: (b3fa653d57300d45) was not able
to be located on http://gpg-keyserver.de/. Upload your public key and try
the operation again.
failureMessage  No public key: Key with id: (b3fa653d57300d45) was not able
to be located on http://pool.sks-keyservers.net:11371. Upload your public
key and try the operation again.
failureMessage  No public key: Key with id: (b3fa653d57300d45) was not able
to be located on http://pgp.mit.edu:11371. Upload your public key and try
the operation again.

Once the close operation returned, I will upload sample files which may
help troubleshoot the issue.

Thanks,

On Sat, Jan 12, 2019 at 9:04 PM Wangda Tan  wrote:

> Thanks David for the quick response!
>
> I just retried, now the "No public key" issue is gone. However,  the
> issue:
>
> failureMessage  Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-tests.jar',
> check the logs.
> failureMessage  Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-test-sources.jar',
> check the logs.
> failureMessage  Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2.pom',
> check the logs.
>
>
> Still exists and repeated hundreds of times. Do you know how to access the
> logs mentioned by above log?
>
> Best,
> Wangda
>
> On Sat, Jan 12, 2019 at 8:37 PM David Nalley  wrote:
>
>> On Sat, Jan 12, 2019 at 9:09 PM Wangda Tan  wrote:
>> >
>> > Hi Devs,
>> >
>> > I'm currently rolling Hadoop 3.1.2 release candidate, however, I saw an
>> issue when I try to close repo in Nexus.
>> >
>> > Logs of https://repository.apache.org/#stagingRepositories
>> (orgapachehadoop-1183) shows hundreds of lines of the following error:
>> >
>> > failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
>> able to be located on http://gpg-keyserver.de/. Upload your public key
>> and try the operation again.
>> > failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
>> able to be located on http://pool.sks-keyservers.net:11371. Upload your
>> public key and try the operation again.
>> > failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
>> able to be located on http://pgp.mit.edu:11371. Upload your public key
>> and try the operation again.
>> > ...
>> > failureMessage  Failed to validate the pgp signature of
>> '/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-tests.jar',
>> check the logs.
>> > failureMessage  Failed to validate the pgp signature of
>> '/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-test-sources.jar',
>> check the logs.
>> > failureMessage  Failed to validate the pgp signature of
>> '/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-sources.jar',
>> check the logs.
>> >
>> >
>> > This is the same key I used before (and finished two releases), the
>> same environment I used before.
>> >
>> > I have tried more than 10 times in the last two days, no luck. And
>> closing the repo takes almost one hour (Regular time is less than 1 min)
>> and always fail at the last.
>> >
>> > I used following commands to validate key exists on key servers
>> >
>> > gpg --keyserver pgp.mit.edu --recv-keys 57300D45
>> > gpg: WARNING: unsafe permissions on homedir '/Users/wtan/.gnupg'
>> > gpg: key B3FA653D57300D45: 1 signature not checked due to a missing key
>> > gpg: key B3FA653D57300D45: "Wangda tan " not changed
>> > gpg: Total number processed: 1
>> > gpg:  unchanged: 1
>> >
>> > gpg --keyserver pool.sks-keyservers.net --recv-keys B3FA653D57300D45
>> > gpg: WARNING: unsafe permissions on homedir '/Users/wtan/.gnupg'
>> > gpg: key B3FA653D57300D45: 1 signature not checked due to a missing key
>> > gpg: key B3FA653D57300D45: "Wangda tan " not changed
>> > gpg: Total number processed: 1
>> > gpg:  unchanged: 1
>> >
>>
>> Both of these report that your key was not found.
>> I took the key from the KEYS file and uploaded it to both of those
>> servers.
>>
>> You might try the release again and see if this resolves the issue.
>>
>

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

Thanks David for the quick response!

I just retried, now the "No public key" issue is gone. However,  the issue:

failureMessage  Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-tests.jar',
check the logs.
failureMessage  Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-test-sources.jar',
check the logs.
failureMessage  Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2.pom',
check the logs.


Still exists and repeated hundreds of times. Do you know how to access the
logs mentioned by above log?

Best,
Wangda

On Sat, Jan 12, 2019 at 8:37 PM David Nalley  wrote:

> On Sat, Jan 12, 2019 at 9:09 PM Wangda Tan  wrote:
> >
> > Hi Devs,
> >
> > I'm currently rolling Hadoop 3.1.2 release candidate, however, I saw an
> issue when I try to close repo in Nexus.
> >
> > Logs of https://repository.apache.org/#stagingRepositories
> (orgapachehadoop-1183) shows hundreds of lines of the following error:
> >
> > failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
> able to be located on http://gpg-keyserver.de/. Upload your public key
> and try the operation again.
> > failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
> able to be located on http://pool.sks-keyservers.net:11371. Upload your
> public key and try the operation again.
> > failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
> able to be located on http://pgp.mit.edu:11371. Upload your public key
> and try the operation again.
> > ...
> > failureMessage  Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-tests.jar',
> check the logs.
> > failureMessage  Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-test-sources.jar',
> check the logs.
> > failureMessage  Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-sources.jar',
> check the logs.
> >
> >
> > This is the same key I used before (and finished two releases), the same
> environment I used before.
> >
> > I have tried more than 10 times in the last two days, no luck. And
> closing the repo takes almost one hour (Regular time is less than 1 min)
> and always fail at the last.
> >
> > I used following commands to validate key exists on key servers
> >
> > gpg --keyserver pgp.mit.edu --recv-keys 57300D45
> > gpg: WARNING: unsafe permissions on homedir '/Users/wtan/.gnupg'
> > gpg: key B3FA653D57300D45: 1 signature not checked due to a missing key
> > gpg: key B3FA653D57300D45: "Wangda tan " not changed
> > gpg: Total number processed: 1
> > gpg:  unchanged: 1
> >
> > gpg --keyserver pool.sks-keyservers.net --recv-keys B3FA653D57300D45
> > gpg: WARNING: unsafe permissions on homedir '/Users/wtan/.gnupg'
> > gpg: key B3FA653D57300D45: 1 signature not checked due to a missing key
> > gpg: key B3FA653D57300D45: "Wangda tan " not changed
> > gpg: Total number processed: 1
> > gpg:  unchanged: 1
> >
>
> Both of these report that your key was not found.
> I took the key from the KEYS file and uploaded it to both of those servers.
>
> You might try the release again and see if this resolves the issue.
>

[Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

Hi Devs,

I'm currently rolling Hadoop 3.1.2 release candidate, however, I saw an
issue when I try to close repo in Nexus.

Logs of https://repository.apache.org/#stagingRepositories
(orgapachehadoop-1183) shows hundreds of lines of the following error:

failureMessage  No public key: Key with id: (b3fa653d57300d45) was not able
to be located on http://gpg-keyserver.de/. Upload your public key and try
the operation again.
failureMessage  No public key: Key with id: (b3fa653d57300d45) was not able
to be located on http://pool.sks-keyservers.net:11371. Upload your public
key and try the operation again.
failureMessage  No public key: Key with id: (b3fa653d57300d45) was not able
to be located on http://pgp.mit.edu:11371. Upload your public key and try
the operation again.
...
failureMessage  Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-tests.jar',
check the logs.
failureMessage  Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-test-sources.jar',
check the logs.
failureMessage  Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-sources.jar',
check the logs.


This is the same key I used before (and finished two releases), the same
environment I used before.

I have tried more than 10 times in the last two days, no luck. And closing
the repo *takes almost one hour* (Regular time is less than 1 min) and
always fail at the last.

I used following commands to validate key exists on key servers

gpg --keyserver pgp.mit.edu --recv-keys 57300D45
gpg: WARNING: unsafe permissions on homedir '/Users/wtan/.gnupg'
gpg: key B3FA653D57300D45: 1 signature not checked due to a missing key
gpg: key B3FA653D57300D45: "Wangda tan " not changed
gpg: Total number processed: 1
gpg:  unchanged: 1

gpg --keyserver pool.sks-keyservers.net --recv-keys B3FA653D57300D45
gpg: WARNING: unsafe permissions on homedir '/Users/wtan/.gnupg'
gpg: key B3FA653D57300D45: 1 signature not checked due to a missing key
gpg: key B3FA653D57300D45: "Wangda tan " not changed
gpg: Total number processed: 1
gpg:  unchanged: 1

Did I miss anything? I also checked Nexus errors page, it is filled by logs
(https://repository.apache.org/service/local/feeds/errorWarning):

2019-01-13 02:04:34 WARN  [etcherImpl-task] -
com.sonatype.central.secure.nexus.plugin.internal.AuthtokenFetcherImpl -
Failed to fetch authtoken: org.apache.http.conn.ConnectTimeoutException:
Connect to secure.central.sonatype.com:443 [
secure.central.sonatype.com/207.223.241.90] failed: connect timed out.


Not sure if it is related to error I saw. Also adding Apache Infra user
email list to rule out Nexus issues.

Hope to get your help sooner if possible.

Thanks,
Wangda .

[NOTICE] branch-3.1.2 created for 3.1.2 release

[Wangda Tan]

Hi devs,

I just created branch-3.1.2 for 3.1.2 release and pushed to remote.
branch-3.1 is still open for new commits required for next 3.1.x release
(which will be 3.1.3). In the meantime, branch-3.1.2 is closed for new
commits unless further noticed. I'm working on RC0 of 3.1.2 release now.

Best,
Wangda

Re: Apache Hadoop 3.1.2 release plan

[Wangda Tan]

Hi all,

Since all blockers have done, I plan to start rolling RC for 3.1.2
tomorrow, please let me know if I missed anything.

Thanks,
Wangda

On Wed, Oct 24, 2018 at 7:30 PM Vinod Kumar Vavilapalli 
wrote:

> 231 fixed JIRAs is already quite a bunch!
>
> I only see 7 JIRAs marked with Affects Version 3.1.2 and only one of them
> as blocker.
>
> Why not just release now as soon as there are no blockers?
>
> Thanks
> +Vinod
>
> > On Oct 24, 2018, at 4:36 PM, Wangda Tan  wrote:
> >
> > Hi, All
> >
> > We have released Apache Hadoop 3.1.1 on Aug 8, 2018. To further
> > improve the quality of the release, I plan to release 3.1.2
> > by Nov. The focus of 3.1.2 will be fixing blockers / critical bugs
> > and other enhancements. So far there are 231 JIRAs [1] have fix
> > version marked to 3.1.2
> >
> > I plan to cut branch-3.1 on Nov 15 and vote for RC on the same day.
> >
> > Please feel free to share your insights.
> >
> > Thanks,
> > Wangda Tan
> >
> > [1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop
> Map/Reduce")
> > AND fixVersion = 3.1.2
>
>

[jira] [Resolved] (YARN-8975) [Submarine] Use predefined Charset object StandardCharsets.UTF_8 instead of String "UTF-8"

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-8975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan resolved YARN-8975.
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.3.0

Committed to trunk, thanks [~tangzhankun] and reviews from [~ajisakaa].

> [Submarine] Use predefined Charset object StandardCharsets.UTF_8 instead of 
> String "UTF-8"
> --
>
> Key: YARN-8975
> URL: https://issues.apache.org/jira/browse/YARN-8975
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Trivial
> Fix For: 3.3.0
>
> Attachments: YARN-8975-trunk.001.patch, YARN-8975-trunk.002.patch
>
>
> {code:java}
> Writer w = new OutputStreamWriter(new FileOutputStream(file), "UTF-8");{code}
> Could be refactored to this to improve a little bit performance due to avoid 
> string lookup:
> {code:java}
> Writer w = new OutputStreamWriter(new FileOutputStream(file), 
> StandardCharsets.UTF_8);{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: New committer: Botong Huang

[Wangda Tan]

Congrats!

Best,
Wangda

On Wed, Nov 21, 2018 at 4:23 PM Srinivas Reddy 
wrote:

> Congratulations Botong !!!
>
> -
> Srinivas
>
> - Typed on tiny keys. pls ignore typos.{mobile app}
>
> On Thu 22 Nov, 2018, 03:27 Chang Qiang Cao 
> > Congrats Botong!
> >
> > On Wed, Nov 21, 2018 at 2:15 PM Subru Krishnan  wrote:
> >
> > > The Project Management Committee (PMC) for Apache Hadoophas invited
> > > Botong Huang to become a committer and we are pleased to announce that
> > > he has accepted.
> > > Being a committer enables easier contribution to theproject since
> > > there is no need to go via the patchsubmission process. This should
> > > enable better productivity.Being a PMC member enables assistance with
> > > the managementand to guide the direction of the project.
> > >
> > > Congrats and welcome aboard.
> > >
> > > -Subru
> > >
> >
>

Re: Apache Hadoop 3.1.2 release plan

[Wangda Tan]

Just cleaned up all tickets and updated tickets which have target version
== 3.1.2 to 3.1.3.

I will roll an RC next Monday, if you have any tickets need to be released
in 3.1.2, please let me know.

Thanks,
Wangda

On Wed, Oct 24, 2018 at 7:30 PM Vinod Kumar Vavilapalli 
wrote:

> 231 fixed JIRAs is already quite a bunch!
>
> I only see 7 JIRAs marked with Affects Version 3.1.2 and only one of them
> as blocker.
>
> Why not just release now as soon as there are no blockers?
>
> Thanks
> +Vinod
>
> > On Oct 24, 2018, at 4:36 PM, Wangda Tan  wrote:
> >
> > Hi, All
> >
> > We have released Apache Hadoop 3.1.1 on Aug 8, 2018. To further
> > improve the quality of the release, I plan to release 3.1.2
> > by Nov. The focus of 3.1.2 will be fixing blockers / critical bugs
> > and other enhancements. So far there are 231 JIRAs [1] have fix
> > version marked to 3.1.2
> >
> > I plan to cut branch-3.1 on Nov 15 and vote for RC on the same day.
> >
> > Please feel free to share your insights.
> >
> > Thanks,
> > Wangda Tan
> >
> > [1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop
> Map/Reduce")
> > AND fixVersion = 3.1.2
>
>

[jira] [Resolved] (YARN-9020) set a wrong AbsoluteCapacity when call ParentQueue#setAbsoluteCapacity

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan resolved YARN-9020.
--
Resolution: Duplicate

Thanks [~jutia] for reporting this. It is a valid issue.

This is dup of YARN-8917, [~Tao Yang] has put a patch already. Closing this as 
dup.

> set a wrong AbsoluteCapacity when call  ParentQueue#setAbsoluteCapacity
> ---
>
> Key: YARN-9020
> URL: https://issues.apache.org/jira/browse/YARN-9020
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: tianjuan
>Assignee: tianjuan
>Priority: Major
>
> set a wrong AbsoluteCapacity when call  ParentQueue#setAbsoluteCapacity
> private void deriveCapacityFromAbsoluteConfigurations(String label,
>  Resource clusterResource, ResourceCalculator rc, CSQueue childQueue) {
> // 3. Update absolute capacity as a float based on parent's minResource and
>  // cluster resource.
>  childQueue.getQueueCapacities().setAbsoluteCapacity(label,
>  (float) childQueue.getQueueCapacities().{color:#d04437}getCapacity(){color}
>  / getQueueCapacities().getAbsoluteCapacity(label));
>  
> {color:#d04437}should be{color} 
> childQueue.getQueueCapacities().setAbsoluteCapacity(label,
>  (float) 
> childQueue.getQueueCapacities().{color:#f6c342}getCapacity(label){color}
>  / getQueueCapacities().getAbsoluteCapacity(label));



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-8993) [Submarine] Add support to run deep learning workload in non-Docker containers

[Wangda Tan (JIRA)]

Wangda Tan created YARN-8993:


 Summary: [Submarine] Add support to run deep learning workload in 
non-Docker containers
 Key: YARN-8993
 URL: https://issues.apache.org/jira/browse/YARN-8993
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Wangda Tan


Now Submarine can well support Docker container, there're some needs to run TF 
w/o Docker container. This JIRA is targeted to support non-docker container 
deep learning workload orchestration.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Run Distributed TensorFlow on YARN

[Wangda Tan]

Hi Robert,

Submarine in 3.2.0 only support Docker container runtime, and in future
releases (maybe 3.2.1), we plan to add support for non-docker containers.

In order to try Submarine, you need to properly configure docker-on-yarn
first.

You can check
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/site/markdown/InstallationScriptEN.md
for installation guide about how to properly setup Docker container on
multiple containers. Submarine embedded an interactive shell to help you
set up this should be straightforward. Added Xun Liu who is the original
author for the installation interactive shell.

Once you get Docker on YARN properly set up, you can follow
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/site/markdown/QuickStart.md
to run the first application.

Also, you can check Submarine slides to better understand how it works.
See: https://www.dropbox.com/s/wuv19b3rt9k2kq6/submarine-v0.pptx?dl=0

Any questions please don't hesitate to let us know.

Thanks,
Wangda



On Thu, Nov 8, 2018 at 10:12 AM Robert Grandl 
wrote:

>  Thanks a lot for your reply.
> Sunil,
> I was trying to follow the steps from:
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/site/markdown/RunningDistributedCifar10TFJobs.md
>
> to run the tensorflow standalone using submarine. I have installed hadoop
> 3.3.0-SNAPSHOT.
> However, when I run the:yarn jar
> path/to/hadoop-yarn-applications-submarine-3.2.0-SNAPSHOT.jar \
>job run --name tf-job-001 --verbose --docker_image
> hadoopsubmarine/tf-1.8.0-gpu:0.0.1 \
>--input_path hdfs://default/dataset/cifar-10-data \
>--env DOCKER_JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/
>--env DOCKER_HADOOP_HDFS_HOME=/hadoop-3.1.0
>--num_workers 1 --worker_resources memory=8G,vcores=2,gpu=1 \
>--worker_launch_cmd "cd /test/models/tutorials/image/cifar10_estimator
> && python cifar10_main.py --data-dir=%input_path%
> --job-dir=%checkpoint_path% --train-steps=1 --eval-batch-size=16
> --train-batch-size=16 --num-gpus=2 --sync" \
>--tensorboard --tensorboard_docker_image wtan/tf-1.8.0-cpu:0.0.3
> command, I get the following error:2018-11-07 21:48:55,831 INFO  [main]
> client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to
> Application History server at /128.105.144.236:10200Exception in thread
> "main" java.lang.IllegalArgumentException: Unacceptable no of cpus
> specified, either zero or negative for component master (or at the global
> level)at
> org.apache.hadoop.yarn.service.utils.ServiceApiUtil.validateServiceResource(ServiceApiUtil.java:457)
>   at
> org.apache.hadoop.yarn.service.utils.ServiceApiUtil.validateComponent(ServiceApiUtil.java:306)
>   at
> org.apache.hadoop.yarn.service.utils.ServiceApiUtil.validateAndResolveService(ServiceApiUtil.java:237)
>   at
> org.apache.hadoop.yarn.service.client.ServiceClient.actionCreate(ServiceClient.java:496)
>   at
> org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.submitJob(YarnServiceJobSubmitter.java:542)
>   at
> org.apache.hadoop.yarn.submarine.client.cli.RunJobCli.run(RunJobCli.java:231)
>   at org.apache.hadoop.yarn.submarine.client.cli.Cli.main(Cli.java:94)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498)at
> org.apache.hadoop.util.RunJar.run(RunJar.java:323)at
> org.apache.hadoop.util.RunJar.main(RunJar.java:236)
>
> It seems that I don't configure somewhere some corresponding resources for
> a master component. However I have a hard time understanding where and what
> to configure. I also looked at the design document you pointed at:
> https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7
>
> and it has a --master_resources flag. However this is not available in
> 3.3.0.
> Could you please advise how to proceed with this?
> Thank you,- Robert
>
> On Tuesday, November 6, 2018, 10:40:20 PM PST, Jonathan Hung <
> jyhung2...@gmail.com> wrote:
>
>  Hi Robert, I also encourage you to check out
> https://github.com/linkedin/TonY (TensorFlow on YARN) which is a platform
> built for this purpose.
>
> Jonathan
> 
> From: Sunil G 
> Sent: Tuesday, November 6, 2018 10:05:14 PM
> To: Robert Grandl
> Cc: yarn-dev@hadoop.apache.org; yarn-dev-h...@hadoop.apache.org; General
> Subject: Re: Run Distributed TensorFlow on YARN
>
> Hi Robert
>
> {Submarine} project helps to run Distributed Tensorflow on top of YARN with
> ease. YARN-8220