date:20171109

[jira] [Resolved] (HADOOP-8555) Incorrect Kerberos configuration

2017-11-09 Thread Andras Bokor (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Bokor resolved HADOOP-8555.
--
Resolution: Invalid

This part of code has been totally changed. This is no longer a valid issue.

> Incorrect Kerberos configuration
> 
>
> Key: HADOOP-8555
> URL: https://issues.apache.org/jira/browse/HADOOP-8555
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.0.0-alpha, 3.0.0-alpha1
>Reporter: Laxman
>  Labels: kerberos, security
>
> When keytab is given ticket cache should not be considered.
> Following configuration tries to use ticket cache even when keytab is 
> configured. We need not configure ticket cache here.
> org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.KerberosConfiguration.getAppConfigurationEntry(String)
> {code}
>   options.put("keyTab", keytab);
>   options.put("principal", principal);
>   options.put("useKeyTab", "true");
>   options.put("storeKey", "true");
>   options.put("doNotPrompt", "true");
>   options.put("useTicketCache", "true");
>   options.put("renewTGT", "true");
>   options.put("refreshKrb5Config", "true");
>   options.put("isInitiator", "false");
>   String ticketCache = System.getenv("KRB5CCNAME");
>   if (ticketCache != null) {
> options.put("ticketCache", ticketCache);
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC0)

2017-11-09 Thread Subru Krishnan

Thanks Vinod for your feedback, we'll incorporate it when we spin RC1.

-Subru/Arun

On Wed, Nov 8, 2017 at 5:34 PM, Vinod Kumar Vavilapalli 
wrote:

> A related point - I thought I mentioned this in one of the release
> preparation threads, but in any case.
>
> Starting 2.7.0, for every .0 release, we've been adding a disclaimer (to
> the voting thread as well as the final release) that the first release can
> potentially go through additional fixes to incompatible changes (besides
> stabilization fixes). We should do this with 2.9.0 too.
>
> This has some history - long before this, we tried two different things:
> (a) downstream projects consume an RC (b) downstream projects consume a
> release. Option (a) was tried many times but it was increasingly getting
> hard to manage this across all the projects that depend on Hadoop. When we
> tried option (b), we used to make .0 as a GA release, but downstream
> projects like Tez, Hive, Spark would come back and find an incompatible
> change - and now we were forced into a conundrum - is fixing this
> incompatible change itself an incompatibility? So to avoid this problem,
> we've started marking the first few releases as alpha eventually making a
> stable point release. Clearly, specific users can still use this in
> production as long as we the Hadoop community reserve the right to fix
> incompatibilities.
>
> Long story short, I'd just add to your voting thread and release notes
> that 2.9.0 still needs to be tested downstream and so users may want to
> wait for subsequent point releases.
>
> Thanks
> +Vinod
>
> > On Nov 8, 2017, at 12:43 AM, Subru Krishnan  wrote:
> >
> > We are canceling the RC due to the issue that Rohith/Sunil identified.
> The
> > issue was difficult to track down as it only happens when you use IP for
> ZK
> > (works fine with host names) and moreover if ZK and RM are co-located on
> > same machine. We are hopeful to get the fix in tomorrow and roll out RC1.
> >
> > Thanks to everyone for the extensive testing/validation. Hopefully cost
> to
> > replicate with RC1 is much lower.
> >
> > -Subru/Arun.
> >
> > On Tue, Nov 7, 2017 at 5:27 PM, Konstantinos Karanasos <
> kkarana...@gmail.com
> >> wrote:
> >
> >> +1 from me too.
> >>
> >> Did the following:
> >> 1) set up a 9-node cluster;
> >> 2) ran some Gridmix jobs;
> >> 3) ran (2) after enabling opportunistic containers (used a mix of
> >> guaranteed and opportunistic containers for each job);
> >> 4) ran (3) but this time enabling distributed scheduling of
> opportunistic
> >> containers.
> >>
> >> All the above worked with no issues.
> >>
> >> Thanks for all the effort guys!
> >>
> >> Konstantinos
> >>
> >>
> >>
> >> Konstantinos
> >>
> >> On Tue, Nov 7, 2017 at 2:56 PM, Eric Badger 
> >> wrote:
> >>
> >>> +1 (non-binding) pending the issue that Sunil/Rohith pointed out
> >>>
> >>> - Verified all hashes and checksums
> >>> - Built from source on macOS 10.12.6, Java 1.8.0u65
> >>> - Deployed a pseudo cluster
> >>> - Ran some example jobs
> >>>
> >>> Thanks,
> >>>
> >>> Eric
> >>>
> >>> On Tue, Nov 7, 2017 at 4:03 PM, Wangda Tan 
> wrote:
> >>>
>  Sunil / Rohith,
> 
>  Could you check if your configs are same as Jonathan posted configs?
>  https://issues.apache.org/jira/browse/YARN-7453?
> >>> focusedCommentId=16242693&
>  page=com.atlassian.jira.plugin.system.issuetabpanels:
>  comment-tabpanel#comment-16242693
> 
>  And could you try if using Jonathan's configs can still reproduce the
>  issue?
> 
>  Thanks,
>  Wangda
> 
> 
>  On Tue, Nov 7, 2017 at 1:52 PM, Arun Suresh 
> >> wrote:
> 
> > Thanks for testing Rohith and Sunil
> >
> > Can you please confirm if it is not a config issue at your end ?
> > We (both Jonathan and myself) just tried testing this on a fresh
> >>> cluster
> > (both automatic and manual) and we are not able to reproduce this.
> >> I've
> > updated the YARN-7453  >> jira/browse/YARN-7453
> 
> > JIRA
> > with details of testing.
> >
> > Cheers
> > -Arun/Subru
> >
> > On Tue, Nov 7, 2017 at 3:17 AM, Rohith Sharma K S <
> > rohithsharm...@apache.org
> >> wrote:
> >
> >> Thanks Sunil for confirmation. Btw, I have raised YARN-7453
> >>  JIRA to track
> >> this
> >> issue.
> >>
> >> - Rohith Sharma K S
> >>
> >> On 7 November 2017 at 16:44, Sunil G  wrote:
> >>
> >>> Hi Subru and Arun.
> >>>
> >>> Thanks for driving 2.9 release. Great work!
> >>>
> >>> I installed cluster built from source.
> >>> - Ran few MR jobs with application priority enabled. Runs fine.
> >>> - Accessed new UI and it also seems fine.
> >>>
> >>> However I am also getting same issue as Rohith reported.

[jira] [Resolved] (HADOOP-14162) Improve release scripts to automate missing steps

2017-11-09 Thread Elek, Marton (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elek, Marton resolved HADOOP-14162.
---
Resolution: Won't Fix

> Improve release scripts to automate missing steps
> -
>
> Key: HADOOP-14162
> URL: https://issues.apache.org/jira/browse/HADOOP-14162
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>
> According to the conversation on the dev mailing list one pain point of the 
> release making is that even with the latest create-release script a lot of 
> steps are not automated.
> This Jira is about creating a script which guides the release manager throw 
> the proces:
> Goals:
>   * It would work even without the apache infrastructure: with custom 
> configuration (forked repositories/alternative nexus), it would be possible 
> to test the scripts even by a non-commiter.  
>   * every step which could be automated should be scripted (create git 
> branches, build,...). if something could be not automated there an 
> explanation could be printed out, and wait for confirmation
>   * Before dangerous steps (eg. bulk jira update) we can ask for confirmation 
> and explain the 
>   * The run should be idempontent (and there should be an option to continue 
> the release from any steps).  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14160) Create dev-support scripts to do the bulk jira update required by the release process

2017-11-09 Thread Elek, Marton (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-14160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elek, Marton resolved HADOOP-14160.
---
Resolution: Won't Fix

> Create dev-support scripts to do the bulk jira update required by the release 
> process
> -
>
> Key: HADOOP-14160
> URL: https://issues.apache.org/jira/browse/HADOOP-14160
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>
> According to the conversation on the dev mailing list one pain point of the 
> release making is the Jira administration.
> This issue is about creating new scripts to 
>  
>  * query apache issue about a possible release (remaining blocking, issues, 
> etc.)
>  * and do bulk changes (eg.  bump fixVersions)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-11-09 Thread Apache Jenkins Server

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/585/

[Nov 8, 2017 9:11:16 AM] (brahma) HDFS-12788. Reset the upload button when file 
upload fails. Contributed
[Nov 8, 2017 10:21:43 AM] (aajisaka) MAPREDUCE-6997. Moving logging APIs over 
to slf4j in
[Nov 8, 2017 10:28:08 AM] (aajisaka) MAPREDUCE-7001. Moving logging APIs over 
to slf4j in
[Nov 8, 2017 4:00:53 PM] (Arun Suresh) YARN-7453. Fix issue where RM fails to 
switch to active after first
[Nov 8, 2017 4:14:02 PM] (Arun Suresh) YARN-7343. Add a junit test for 
ContainerScheduler recovery. (Sampada
[Nov 9, 2017 12:43:49 AM] (templedf) YARN-7166. Container REST endpoints should 
report resource types
[Nov 9, 2017 1:31:14 AM] (templedf) YARN-7458. TestContainerManagerSecurity is 
still flakey (Contributed by
[Nov 9, 2017 2:07:12 AM] (subru) HADOOP-15025. Ensure singleton for 
ResourceEstimatorService. (Rui Li via
[Nov 9, 2017 6:14:46 AM] (aajisaka) HDFS-12732. Correct spellings of ramdomly 
to randomly in log.




-1 overall


The following subsystems voted -1:
asflicense findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
   org.apache.hadoop.yarn.api.records.Resource.getResources() may expose 
internal representation by returning Resource.resources At Resource.java:by 
returning Resource.resources At Resource.java:[line 215] 

Unreaped Processes :

   hadoop-mapreduce-client-jobclient:1 

Failed junit tests :

   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting 
   
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery 
   hadoop.hdfs.TestReadStripedFileWithMissingBlocks 
   hadoop.yarn.server.TestDiskFailures 
   hadoop.mapreduce.lib.join.TestJoinDatamerge 
   hadoop.mapred.lib.TestDelegatingInputFormat 
   hadoop.mapreduce.lib.input.TestDelegatingInputFormat 
   hadoop.mapred.join.TestDatamerge 
   hadoop.mapreduce.lib.join.TestJoinProperties 
   hadoop.streaming.TestMultipleArchiveFiles 
   hadoop.streaming.TestMultipleCachefiles 
   hadoop.streaming.TestSymLink 
   hadoop.contrib.utils.join.TestDataJoin 

Timed out junit tests :

   org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands 
   
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA 
   
org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA 
   
org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA 
   org.apache.hadoop.mapred.pipes.TestPipeApplication 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/585/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/585/artifact/out/diff-compile-javac-root.txt
  [276K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/585/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/585/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/585/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/585/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/585/artifact/out/whitespace-eol.txt
  [8.8M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/585/artifact/out/whitespace-tabs.txt
  [288K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/585/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/585/artifact/out/diff-javadoc-javadoc-root.txt
  [760K]

   UnreapedProcessesLog:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/585/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-reaper.txt
  [4.0K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/585/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [300K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/585/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [64K]

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC0)

2017-11-09 Thread Chris Douglas

The labor required for these release formalisms is exceeding their
value. Our minor releases have more bugs than our patch releases (we
hope), but every consumer should understand how software versioning
works. Every device I own has bugs on major OS updates. That doesn't
imply that every minor release is strictly less stable than a patch
release, and users need to be warned off it.

In contrast, we should warn users about features that compromise
invariants like security or durability, either by design or due to
their early stage of development. We can't reasonably expect them to
understand those tradeoffs, since they depend on internal details of
Hadoop.

On Wed, Nov 8, 2017 at 5:34 PM, Vinod Kumar Vavilapalli
 wrote:
> When we tried option (b), we used to make .0 as a GA release, but downstream 
> projects like Tez, Hive, Spark would come back and find an incompatible 
> change - and now we were forced into a conundrum - is fixing this 
> incompatible change itself an incompatibility?

Every project takes these case-by-case. Most of the time we'll
accommodate the old semantics- and we try to be explicit where we
promise compatibility- but this isn't a logic problem, it's a
practical one. If it's an easy fix to an obscure API, we probably
won't even hear about it.

> Long story short, I'd just add to your voting thread and release notes that 
> 2.9.0 still needs to be tested downstream and so users may want to wait for 
> subsequent point releases.

It's uncomfortable to have four active release branches, with 3.1
coming in early 2018. We all benefit from the shared deployment
experiences that harden these releases, and fragmentation creates
incentives to compete for that attention. Rather than tacitly
scuffling over waning interest in the 2.x series, I'd endorse your
other thread encouraging consolidation around 3.x.

To that end, there is no policy or precedent that requires that new
minor releases be labeled as "alpha". If there is cause to believe
that 2.9.0 is not ready to release in the stable line, then we
shouldn't release it. -C

>> On Nov 8, 2017, at 12:43 AM, Subru Krishnan  wrote:
>>
>> We are canceling the RC due to the issue that Rohith/Sunil identified. The
>> issue was difficult to track down as it only happens when you use IP for ZK
>> (works fine with host names) and moreover if ZK and RM are co-located on
>> same machine. We are hopeful to get the fix in tomorrow and roll out RC1.
>>
>> Thanks to everyone for the extensive testing/validation. Hopefully cost to
>> replicate with RC1 is much lower.
>>
>> -Subru/Arun.
>>
>> On Tue, Nov 7, 2017 at 5:27 PM, Konstantinos Karanasos >> wrote:
>>
>>> +1 from me too.
>>>
>>> Did the following:
>>> 1) set up a 9-node cluster;
>>> 2) ran some Gridmix jobs;
>>> 3) ran (2) after enabling opportunistic containers (used a mix of
>>> guaranteed and opportunistic containers for each job);
>>> 4) ran (3) but this time enabling distributed scheduling of opportunistic
>>> containers.
>>>
>>> All the above worked with no issues.
>>>
>>> Thanks for all the effort guys!
>>>
>>> Konstantinos
>>>
>>>
>>>
>>> Konstantinos
>>>
>>> On Tue, Nov 7, 2017 at 2:56 PM, Eric Badger 
>>> wrote:
>>>
 +1 (non-binding) pending the issue that Sunil/Rohith pointed out

 - Verified all hashes and checksums
 - Built from source on macOS 10.12.6, Java 1.8.0u65
 - Deployed a pseudo cluster
 - Ran some example jobs

 Thanks,

 Eric

 On Tue, Nov 7, 2017 at 4:03 PM, Wangda Tan  wrote:

> Sunil / Rohith,
>
> Could you check if your configs are same as Jonathan posted configs?
> https://issues.apache.org/jira/browse/YARN-7453?
 focusedCommentId=16242693&
> page=com.atlassian.jira.plugin.system.issuetabpanels:
> comment-tabpanel#comment-16242693
>
> And could you try if using Jonathan's configs can still reproduce the
> issue?
>
> Thanks,
> Wangda
>
>
> On Tue, Nov 7, 2017 at 1:52 PM, Arun Suresh 
>>> wrote:
>
>> Thanks for testing Rohith and Sunil
>>
>> Can you please confirm if it is not a config issue at your end ?
>> We (both Jonathan and myself) just tried testing this on a fresh
 cluster
>> (both automatic and manual) and we are not able to reproduce this.
>>> I've
>> updated the YARN-7453 >> jira/browse/YARN-7453
>
>> JIRA
>> with details of testing.
>>
>> Cheers
>> -Arun/Subru
>>
>> On Tue, Nov 7, 2017 at 3:17 AM, Rohith Sharma K S <
>> rohithsharm...@apache.org
>>> wrote:
>>
>>> Thanks Sunil for confirmation. Btw, I have raised YARN-7453
>>>  JIRA to track
>>> this
>>> issue.
>>>
>>> - Rohith Sharma K S
>>>
>>> On 7 November 2017 at 16:44, Sunil G

[jira] [Resolved] (HADOOP-15012) Add readahead, dropbehind, and unbuffer to StreamCapabilities

2017-11-09 Thread John Zhuge (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge resolved HADOOP-15012.
-
   Resolution: Fixed
Fix Version/s: 3.1.0

Committed to trunk together with HADOOP-14872. Code review was done there.
{noformat}
6c32ddad302 HADOOP-14872. CryptoInputStream should implement unbuffer. 
Contributed by John Zhuge.
bf6a660232b HADOOP-15012. Add readahead, dropbehind, and unbuffer to 
StreamCapabilities. Contributed by John Zhuge.
{noformat}


> Add readahead, dropbehind, and unbuffer to StreamCapabilities
> -
>
> Key: HADOOP-15012
> URL: https://issues.apache.org/jira/browse/HADOOP-15012
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.9.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Fix For: 3.1.0
>
>
> A split from HADOOP-14872 to track changes that enhance StreamCapabilities 
> class with READAHEAD, DROPBEHIND, and UNBUFFER capability.
> Discussions and code reviews are done in HADOOP-14872.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-7410) Mavenize common RPM/DEB

2017-11-09 Thread Allen Wittenauer (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-7410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HADOOP-7410.
--
Resolution: Won't Fix

> Mavenize common RPM/DEB
> ---
>
> Key: HADOOP-7410
> URL: https://issues.apache.org/jira/browse/HADOOP-7410
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Alejandro Abdelnur
>Assignee: Eric Yang
>
> Mavenize RPM/DEB generation



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[VOTE] Release Apache Hadoop 2.9.0 (RC1)

2017-11-09 Thread Subru Krishnan

Hi Folks,

Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be the
starting release for Apache Hadoop 2.9.x line - it includes 30 New Features
with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
2.8.2 .

More information about the 2.9.0 release plan can be found here:
*https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
*

New RC is available at: http://home.apache.org/~asuresh/hadoop-2.9.0-RC1/


The RC tag in git is: release-2.9.0-RC1, and the latest commit id is:
7d2ba3e8dd74d2631c51ce6790d59e50eeb7a846.

The maven artifacts are available via repository.apache.org at:
https://repository.apache.org/content/repositories/orgapachehadoop-1066


Please try the release and vote; the vote will run for the usual 5 days,
ending on Tuesday 14th November 2017 6pm PST time.

Thanks,
-Subru/Arun

[YARN-6483] Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes returned by the Resource Manager as a response to the Application Master heartbeat

2017-11-09 Thread Juan Rodríguez Hortalá

Hi,

I have submitted a pull request https://github.com/apache/hadoop/pull/289
 for https://issues.apache.org/jira/browse/YARN-6483. This change is an
alternative proposal to YARN-3224, for notifying application masters about
the decommission of a node, that could help completing
https://issues.apache.org/jira/browse/YARN-914. This could be useful for
implementing functionality like for example
https://github.com/apache/spark/pull/19267, where a Spark AM is able to
blacklist executors when their corresponding node transitions into
DECOMMISSIONING state.

I would like to hear your feedback on this.

Thanks,

Juan Rodriguez Hortala

RE: [VOTE] Merge yarn-native-services branch into trunk

2017-11-09 Thread Zheng, Kai

Cool to have this feature! Thanks Jian and all.

Regards,
Kai

-Original Message-
From: Vinod Kumar Vavilapalli [mailto:vino...@apache.org] 
Sent: Tuesday, November 07, 2017 7:20 AM
To: Jian He 
Cc: yarn-...@hadoop.apache.org; common-dev@hadoop.apache.org; Hdfs-dev 
; mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Merge yarn-native-services branch into trunk

Congratulations to all the contributors involved, this is a great step forward!

+Vinod

> On Nov 6, 2017, at 2:40 PM, Jian He  wrote:
> 
> Okay, I just merged the branch to trunk (108 commits in total !) 
> Again, thanks for all who contributed to this feature!
> 
> Jian
> 
> On Nov 6, 2017, at 1:26 PM, Jian He 
> > wrote:
> 
> Here’s +1 from myself.
> The vote passes with 7 (+1) bindings and 2 (+1) non-bindings.
> 
> Thanks for all who voted. I’ll merge to trunk by the end of today.
> 
> Jian
> 
> On Nov 6, 2017, at 8:38 AM, Billie Rinaldi 
> > wrote:
> 
> +1 (binding)
> 
> On Mon, Oct 30, 2017 at 1:06 PM, Jian He 
> > wrote:
> Hi All,
> 
> I would like to restart the vote for merging yarn-native-services to trunk.
> Since last vote, we have been working on several issues in documentation, 
> DNS, CLI modifications etc. We believe now the feature is in a much better 
> shape.
> 
> Some back ground:
> At a high level, the following are the key feautres implemented.
> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate 
> existing services to YARN either docker or non-docker based.
> - YARN-4793[2]. A Rest API service embeded in RM (optional)  for user 
> to deploy a service via a simple JSON spec
> - YARN-4757[3]. Extending today's service registry with a simple DNS 
> service to enable users to discover services deployed on YARN via 
> standard DNS lookup
> - YARN-6419[4]. UI support for native-services on the new YARN UI All 
> these new services are optional and are sitting outside of the existing 
> system, and have no impact on existing system if disabled.
> 
> Special thanks to a team of folks who worked hard towards this: Billie 
> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma K 
> S, Sunil G, Akhil PB, Eric Yang. This effort could not be possible without 
> their ideas and hard work.
> Also thanks Allen for some review and verifications.
> 
> Thanks,
> Jian
> 
> [1] https://issues.apache.org/jira/browse/YARN-5079
> [2] https://issues.apache.org/jira/browse/YARN-4793
> [3] https://issues.apache.org/jira/browse/YARN-4757
> [4] https://issues.apache.org/jira/browse/YARN-6419
> 
> 
> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15027) Improvements for Hadoop read from AliyunOSS

2017-11-09 Thread wujinhu (JIRA)

wujinhu created HADOOP-15027:


 Summary: Improvements for Hadoop read from AliyunOSS
 Key: HADOOP-15027
 URL: https://issues.apache.org/jira/browse/HADOOP-15027
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs/oss
Reporter: wujinhu


Currently, read performance is poor when Hadoop reads from AliyunOSS. It needs 
about 1min to read 1GB from OSS.
Class AliyunOSSInputStream uses single thread to read data from AliyunOSS,  so 
we can refactor this by using multi-thread pre read to improve this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC1)

2017-11-09 Thread John Zhuge

+1 (binding)

   - Verified checksums and signatures of all tarballs
   - Built source with native, Java 1.8.0_131-b11 on Mac OS X 10.12.6
   - Verified these cloud connectors:
  - All S3A integration tests
  - All ADL live unit tests
   - Deployed both binary and built source to a pseudo cluster, passed the
   following sanity tests in insecure, SSL, and SSL+Kerberos mode:
  - HDFS basic and ACL
  - DistCp basic
  - MapReduce wordcount (skipped in SSL+Kerberos mode)
  - KMS and HttpFS basic
  - Balancer start/stop


On Thu, Nov 9, 2017 at 5:39 PM, Subru Krishnan  wrote:

> Hi Folks,
>
> Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be the
> starting release for Apache Hadoop 2.9.x line - it includes 30 New Features
> with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
> 2.8.2 .
>
> More information about the 2.9.0 release plan can be found here:
> *https://cwiki.apache.org/confluence/display/HADOOP/
> Roadmap#Roadmap-Version2.9
>  Roadmap#Roadmap-Version2.9>*
>
> New RC is available at: http://home.apache.org/~asuresh/hadoop-2.9.0-RC1/
>  2F~asuresh%2Fhadoop-2.9.0-RC1%2F=D=1=
> AFQjCNE7BF35IDIMZID3hPqiNglWEVsTpg>
>
> The RC tag in git is: release-2.9.0-RC1, and the latest commit id is:
> 7d2ba3e8dd74d2631c51ce6790d59e50eeb7a846.
>
> The maven artifacts are available via repository.apache.org at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1066
>  apache.org%2Fcontent%2Frepositories%2Forgapachehadoop-1066=D&
> sntz=1=AFQjCNFcern4uingMV_sEreko_zeLlgdlg>
>
> Please try the release and vote; the vote will run for the usual 5 days,
> ending on Tuesday 14th November 2017 6pm PST time.
>
> Thanks,
> -Subru/Arun
>



-- 
John

[jira] [Resolved] (HADOOP-8555) Incorrect Kerberos configuration

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC0)

[jira] [Resolved] (HADOOP-14162) Improve release scripts to automate missing steps

[jira] [Resolved] (HADOOP-14160) Create dev-support scripts to do the bulk jira update required by the release process

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC0)

[jira] [Resolved] (HADOOP-15012) Add readahead, dropbehind, and unbuffer to StreamCapabilities

[jira] [Resolved] (HADOOP-7410) Mavenize common RPM/DEB

[VOTE] Release Apache Hadoop 2.9.0 (RC1)

[YARN-6483] Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes returned by the Resource Manager as a response to the Application Master heartbeat

RE: [VOTE] Merge yarn-native-services branch into trunk

[jira] [Created] (HADOOP-15027) Improvements for Hadoop read from AliyunOSS

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC1)

13 matches

Site Navigation

Mail list logo

Footer information