[jira] [Created] (HADOOP-14744) RM and NM killed automatically

[Manish (JIRA)]

Manish created HADOOP-14744:
---

 Summary: RM and NM killed automatically
 Key: HADOOP-14744
 URL: https://issues.apache.org/jira/browse/HADOOP-14744
 Project: Hadoop Common
  Issue Type: Improvement
  Components: common
Affects Versions: 2.6.0
 Environment: We have Apache hadoop setup with Zookeeper enabled.
Reporter: Manish


We have Apache hadoop setup with Zookeeper enabled.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14742) Document multi-URI replication Inode for ViewFS

[Chris Douglas (JIRA)]

Chris Douglas created HADOOP-14742:
--

 Summary: Document multi-URI replication Inode for ViewFS
 Key: HADOOP-14742
 URL: https://issues.apache.org/jira/browse/HADOOP-14742
 Project: Hadoop Common
  Issue Type: Task
  Components: documentation, viewfs
Affects Versions: 3.0.0-beta1
Reporter: Chris Douglas


HADOOP-12077 added client-side "replication" capabilities to ViewFS. Its 
semantics and configuration should be documented.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14418) Confusing failure stack trace when codec fallback is happend

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HADOOP-14418.
--
Resolution: Duplicate

Thanks [~lewuathe] for the work here!

> Confusing failure stack trace when codec fallback is happend
> 
>
> Key: HADOOP-14418
> URL: https://issues.apache.org/jira/browse/HADOOP-14418
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
>Priority: Minor
>  Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HADOOP-14418.01.patch, HADOOP-14418.02.patch
>
>
> When erasure codec is fallbacked, all stacktrace is shown to client.
> {code}
> root@990705591ccc:/usr/local/hadoop# bin/hadoop fs -put README.txt /ec
> 17/05/13 08:23:46 WARN util.NativeCodeLoader: Unable to load native-hadoop 
> library for your platform... using builtin-java classes where applicable
> 17/05/13 08:23:47 WARN erasurecode.CodecUtil: Failed to create raw erasure 
> encoder rs_native, fallback to next codec if possible
> java.lang.ExceptionInInitializerError
>   at 
> org.apache.hadoop.io.erasurecode.rawcoder.NativeRSRawErasureCoderFactory.createEncoder(NativeRSRawErasureCoderFactory.java:35)
>   at 
> org.apache.hadoop.io.erasurecode.CodecUtil.createRawEncoderWithFallback(CodecUtil.java:173)
>   at 
> org.apache.hadoop.io.erasurecode.CodecUtil.createRawEncoder(CodecUtil.java:129)
>   at 
> org.apache.hadoop.hdfs.DFSStripedOutputStream.(DFSStripedOutputStream.java:302)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:309)
>   at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1214)
>   at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1193)
>   at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1131)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:449)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:446)
>   at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:460)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
>   at 
> org.apache.hadoop.fs.FilterFileSystem.create(FilterFileSystem.java:181)
>   at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1074)
>   at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1054)
>   at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:943)
>   at 
> org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.create(CommandWithDestination.java:509)
>   at 
> org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:484)
>   at 
> org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:407)
>   at 
> org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:342)
>   at 
> org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:277)
>   at 
> org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:262)
>   at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331)
>   at 
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:303)
>   at 
> org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:257)
>   at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:285)
>   at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:269)
>   at 
> org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:228)
>   at 
> org.apache.hadoop.fs.shell.CopyCommands$Put.processArguments(CopyCommands.java:286)
>   at 
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:119)
>   at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
>   at org.apache.hadoop.fs.FsShell.run(FsShell.java:326)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
>   at org.apache.hadoop.fs.FsShell.main(FsShell.java:389)
> Caused by: java.lang.RuntimeException: hadoop native library cannot be loaded.
>   at 
> org.apache.hadoop.io.erasurecode.ErasureCodeNative.checkNativeCodeLoaded(ErasureCodeNative.java:69)
>   at 
> org.apache.hadoop.io.erasurecode.rawcoder.NativeRSRawEncoder.(NativeRSRawEncoder.java:33)
>   ... 36 more
> root@990705591ccc:/usr/local/hadoop#
> {code}
> This message is confusing to 

Re: [RESULT] [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Zhe Zhang]

Thanks Konstantin, great work!

On Sat, Aug 5, 2017 at 1:36 PM Konstantin Shvachko 
wrote:

> My formal vote
> +1 (binding)
>
> I am glad to summarize that
> with 7 binding and 13 non-binding +1s and no -1s the vote for Apache
> Release 2.7.4 passes.
> Thank you everybody for contributing to the release, testing it, and
> voting.
>
> Binding +1s (7)
> Zhe Zhang
> Jason Lowe
> Eric Payne
> Sunil G
> Akira Ajisaka
> Chris Douglas
> Konstantin Shvachko
>
> Non-binding +1s (13)
> John Zhuge
> Surendra Lilhore
> Masatake Iwasaki
> Hanisha Koneru
> Chen Liang
> Fnu Ajay Kumar
> Brahma Reddy Battula
> Edwina Lu
> Ye Zhou
> Eric Badger
> Mingliang Liu
> Kuhu Shukla
> Erik Krogen
>
> Thanks,
> --Konstantin
>
> On Sat, Jul 29, 2017 at 4:29 PM, Konstantin Shvachko  >
> wrote:
>
> > Hi everybody,
> >
> > Here is the next release of Apache Hadoop 2.7 line. The previous stable
> > release 2.7.3 was available since 25 August, 2016.
> > Release 2.7.4 includes 264 issues fixed after release 2.7.3, which are
> > critical bug fixes and major optimizations. See more details in Release
> > Note:
> > http://home.apache.org/~shv/hadoop-2.7.4-RC0/releasenotes.html
> >
> > The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.4-RC0/
> >
> > Please give it a try and vote on this thread. The vote will run for 5
> days
> > ending 08/04/2017.
> >
> > Please note that my up to date public key are available from:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > Please don't forget to refresh the page if you've been there recently.
> > There are other place on Apache sites, which may contain my outdated key.
> >
> > Thanks,
> > --Konstantin
> >
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: Question about how to best contribute

[John Zhuge]

And check out HADOOP-12145
 Organize and update
CodeReviewChecklist wiki.

Thanks, your contribution will be greatly appreciated!


On Mon, Aug 7, 2017 at 5:53 AM, Steve Loughran 
wrote:

>
> Hi Lars & Welcome!
>
> Maybe the first step here would be look at those style guides and think
> how to bring them up to date, especially with stuff like lambda-expressions
> in java 8, and mnodules forthcoming in in java 9, SLF4J logging, Junit 5 ->
> 5 testing, code instrumentation, diagnostics, log stability, etc.
>
> https://issues.apache.org/jira/browse/HADOOP-12143 . ;
>
> This is my go at doing this
>
> https://github.com/steveloughran/formality/blob/
> master/styleguide/styleguide.md
>
>
> I've not done any work on trying to get it in, more evolving it as how I
> code & what I look for, especially in tests.
>
> If you want to take this on, it'd be nice. At the same time, I fear
> there'd be push back if you turned up and started telling people what to
> do. Collaborating with us all on the test code is a good place to start.
>
> We're also more relaxed about contributions to the less-core bits of the
> system (things like HDFS, IPC, security and Yarn core are trouble). If
> there's stuff outside that you want to take a go at helping clean up,
> that'd be lower risk (example: object store connectors)
>
> -Steve
>
>
>
> On 7 Aug 2017, at 13:13, Lars Francke > wrote:
>
> Hi,
>
> a few words about me: I've contributed to Hadoop (and it's ecosystem[4]) in
> the past am a Hive committer and have used Hadoop for 10 years now, so I'm
> not totally inexperienced. I'm earning my money as a Hadoop consultant so
> I've seen dozens of real-life clusters in my life.
>
> As part of a few recent client projects and now writing about Hadoop in a
> new project/book I'm digging into the source code to figure out some of the
> things that are not documented.
>
> But as part of this digging I'm seeing lots of warnings in the code,
> inconsistencies etc. and I'd like to contribute some fixes to this back to
> the community.
>
> I have been a long-time believer in good code quality and consistent code
> styles. This might affect people like me especially who do a lot of
> "drive-by" contributions as I'm not someone who looks at the code daily but
> comes across it reasonably often as part of client engagements. In those
> scenarios, it's very unhelpful to have inconsistent code & bad
> documentation.
>
> Two simple but concrete examples:
> * There's lots of "final" usages on variables and methods but no
> consistency. Was this done for particular reasons or personal preference?
>
> personal, though with a move to l-expressions, it matters a lot more. We
> should really be marking all parameters as final at the very least.
>
>
> * Similarly, there's lots of things that are public or protected while they
> could in theory be private. This especially makes it very hard to reason
> about code.
>
> there's now a bit of fear of breaking things, but at the very least,
> things could be protected or package-private more than they are.
>
>
>
> Judging from the current code there's lots of "unofficial" code styling
> and/or personal preference. The Wiki says[1] to follow the Sun
> guidelines[2] which have not been updated in almost 20 years. A new version
> is in the works an clarifies a lot of things[3]. I'm trying to get it
> published soon. I'd try to format according to the latter (that means among
> other things no "final" for local variables).
>
> I realize that I won't be able to single-handedly fix all of this
> especially as code gets contributed but if the community thinks it's
> worthwhile I'd still love to land a few cleanup patches. My experience in
> the past has been that it's hard to get attention to these things (which I
> fully understand as they take up someone's time to review & commit).
>
> So, this is my request for comments on these questions:
> * Is there any interest in this at all?
> ** "This" being patches for code style & things like FindBugs & Checkstyle
> warnings
> * Size of the patches: Rather one big patch or smaller ones (e.g. per file
> or package)
> * Anyone willing to help me with this? e.g. reviewing and committing? I'd
> be more than happy to bribe you with drinks, sweets, food or something else
>
> My plan is not to go through each and every file and fix every issue I see.
> But there are some specific areas I'm looking at in detail and there I'd
> love to contribute back.
>
> Thank you for reading!
>
> Cheers,
> Lars
>
> PS: Posting to common-dev only, not sure if I should cross post to hdfs-dev
> and yarn-dev as well?
>
> [1] 
> [2] <
> http://www.oracle.com/technetwork/java/javase/documentation/codeconvtoc-
> 136057.html
>
> [3] 
> [4] <
> 

[jira] [Created] (HADOOP-14741) Refactor curator based ZooKeeper communication into common library

[Subru Krishnan (JIRA)]

Subru Krishnan created HADOOP-14741:
---

 Summary: Refactor curator based ZooKeeper communication into 
common library
 Key: HADOOP-14741
 URL: https://issues.apache.org/jira/browse/HADOOP-14741
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Subru Krishnan
Assignee: Inigo Goiri


Currently we have ZooKeeper based store implementations for multiple state 
stores like RM, YARN Federation, HDFS router-based federation, RM queue configs 
etc. This jira proposes to unify the curator based ZK communication to 
eliminate redundancies.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Integration Tests build line

[Ewan Higgs]

Hi all,
I’m having trouble getting integration tests to run successfully. It hasn’t 
really affected me so far, but it’s bothering me that I’m not getting these 
working and I’d like this to work. So maybe I’m doing something wrong. I tried 
looking in Jenkins but I didn’t see the integration tests being used anywhere.


mvn install -Dtest=**/ITUseMiniCluster.java -DforkMode=never -Pnoshade

Running org.apache.hadoop.example.ITUseMiniCluster
Tests run: 4, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 35.641 sec <<< 
FAILURE! - in org.apache.hadoop.example.ITUseMiniCluster
useWebHDFS(org.apache.hadoop.example.ITUseMiniCluster)  Time elapsed: 21.097 
sec  <<< ERROR!
java.lang.NoClassDefFoundError: 
org/apache/hadoop/shaded/org/mockito/stubbing/Answer
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at 
org.apache.maven.surefire.booter.IsolatedClassLoader.loadClass(IsolatedClassLoader.java:97)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.shouldWait(MiniDFSCluster.java:2668)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.waitActive(MiniDFSCluster.java:2564)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.waitActive(MiniDFSCluster.java:2607)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1667)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:874)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:494)
at 
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:453)
at 
org.apache.hadoop.example.ITUseMiniCluster.clusterUp(ITUseMiniCluster.java:74)

useWebHDFS(org.apache.hadoop.example.ITUseMiniCluster)  Time elapsed: 21.1 sec  
<<< ERROR!
java.lang.NullPointerException: null
at 
org.apache.hadoop.example.ITUseMiniCluster.clusterDown(ITUseMiniCluster.java:80)

useHdfsFileSystem(org.apache.hadoop.example.ITUseMiniCluster)  Time elapsed: 
14.486 sec  <<< ERROR!
java.lang.NoClassDefFoundError: 
org/apache/hadoop/shaded/org/mockito/stubbing/Answer
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at 
org.apache.maven.surefire.booter.IsolatedClassLoader.loadClass(IsolatedClassLoader.java:97)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.shouldWait(MiniDFSCluster.java:2668)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.waitActive(MiniDFSCluster.java:2564)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.waitActive(MiniDFSCluster.java:2607)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1667)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:874)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:494)
at 
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:453)
at 
org.apache.hadoop.example.ITUseMiniCluster.clusterUp(ITUseMiniCluster.java:74)

useHdfsFileSystem(org.apache.hadoop.example.ITUseMiniCluster)  Time elapsed: 
14.486 sec  <<< ERROR!
java.lang.NullPointerException: null
at 
org.apache.hadoop.example.ITUseMiniCluster.clusterDown(ITUseMiniCluster.java:80)



Thanks for any help!

-Ewan

Western Digital Corporation (and its subsidiaries) E-mail Confidentiality 
Notice & Disclaimer:

This e-mail and any files transmitted with it may contain confidential or 
legally privileged information of WDC and/or its affiliates, and are intended 
solely for the use of the individual or entity to which they are addressed. If 
you are not the intended recipient, any disclosure, copying, distribution or 
any action taken or omitted to be taken in reliance on it, is prohibited. If 
you have received this e-mail in error, please notify the sender immediately 
and delete the e-mail in its entirety from your system.

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/485/

[Aug 6, 2017 7:19:23 PM] (stevel) HADOOP-14722. Azure: BlockBlobInputStream 
position incorrect after seek.




-1 overall


The following subsystems voted -1:
findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-hdfs-project/hadoop-hdfs-client 
   Possible exposure of partially initialized object in 
org.apache.hadoop.hdfs.DFSClient.initThreadsNumForStripedReads(int) At 
DFSClient.java:object in 
org.apache.hadoop.hdfs.DFSClient.initThreadsNumForStripedReads(int) At 
DFSClient.java:[line 2906] 
   org.apache.hadoop.hdfs.server.protocol.SlowDiskReports.equals(Object) 
makes inefficient use of keySet iterator instead of entrySet iterator At 
SlowDiskReports.java:keySet iterator instead of entrySet iterator At 
SlowDiskReports.java:[line 105] 

FindBugs :

   module:hadoop-hdfs-project/hadoop-hdfs 
   Possible null pointer dereference in 
org.apache.hadoop.hdfs.qjournal.server.JournalNode.getJournalsStatus() due to 
return value of called method Dereferenced at 
JournalNode.java:org.apache.hadoop.hdfs.qjournal.server.JournalNode.getJournalsStatus()
 due to return value of called method Dereferenced at JournalNode.java:[line 
302] 
   
org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setClusterId(String)
 unconditionally sets the field clusterId At HdfsServerConstants.java:clusterId 
At HdfsServerConstants.java:[line 193] 
   
org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setForce(int)
 unconditionally sets the field force At HdfsServerConstants.java:force At 
HdfsServerConstants.java:[line 217] 
   
org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setForceFormat(boolean)
 unconditionally sets the field isForceFormat At 
HdfsServerConstants.java:isForceFormat At HdfsServerConstants.java:[line 229] 
   
org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setInteractiveFormat(boolean)
 unconditionally sets the field isInteractiveFormat At 
HdfsServerConstants.java:isInteractiveFormat At HdfsServerConstants.java:[line 
237] 
   Possible null pointer dereference in 
org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocksHelper(File, File, 
int, HardLink, boolean, File, List) due to return value of called method 
Dereferenced at 
DataStorage.java:org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocksHelper(File,
 File, int, HardLink, boolean, File, List) due to return value of called method 
Dereferenced at DataStorage.java:[line 1339] 
   Possible null pointer dereference in 
org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldLegacyOIVImages(String,
 long) due to return value of called method Dereferenced at 
NNStorageRetentionManager.java:org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldLegacyOIVImages(String,
 long) due to return value of called method Dereferenced at 
NNStorageRetentionManager.java:[line 258] 
   Useless condition:argv.length >= 1 at this point At DFSAdmin.java:[line 
2100] 
   Useless condition:numBlocks == -1 at this point At 
ImageLoaderCurrent.java:[line 727] 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 
   Useless object stored in variable removedNullContainers of method 
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List)
 At NodeStatusUpdaterImpl.java:removedNullContainers of method 
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List)
 At NodeStatusUpdaterImpl.java:[line 642] 
   
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeVeryOldStoppedContainersFromCache()
 makes inefficient use of keySet iterator instead of entrySet iterator At 
NodeStatusUpdaterImpl.java:keySet iterator instead of entrySet iterator At 
NodeStatusUpdaterImpl.java:[line 719] 
   Hard coded reference to an absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:[line 490] 
   
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.createStatus()
 makes inefficient use of keySet iterator instead of entrySet iterator At 
ContainerLocalizer.java:keySet iterator 

[jira] [Created] (HADOOP-14740) KMSJsonReader fails when no payload is provided

[Lars Francke (JIRA)]

Lars Francke created HADOOP-14740:
-

 Summary: KMSJsonReader fails when no payload is provided
 Key: HADOOP-14740
 URL: https://issues.apache.org/jira/browse/HADOOP-14740
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Lars Francke
Priority: Minor


When using one of the {{POST}} operations (e.g. rollover a key) and not 
actually providing a JSON payload the {{KMSJsonReader}} fails with this error 
message:

{quote}No content to map due to end-of-input{quote}

I think the only method affected today might be the rollover operation but it's 
still a bug and might affect other new operations added later on.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: HADOOP-14163 proposal for new hadoop.apache.org

[Allen Wittenauer]

> On Aug 7, 2017, at 3:53 AM, Akira Ajisaka  wrote:
> 
> 
> I'll ask INFRA to create a git repository if there are no objections.

There's no need to create a git repo.  They just need to know to pull 
the website from the asf-site branch.
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Question about how to best contribute

[Steve Loughran]

Hi Lars & Welcome!

Maybe the first step here would be look at those style guides and think how to 
bring them up to date, especially with stuff like lambda-expressions in java 8, 
and mnodules forthcoming in in java 9, SLF4J logging, Junit 5 -> 5 testing, 
code instrumentation, diagnostics, log stability, etc.

https://issues.apache.org/jira/browse/HADOOP-12143 . ;

This is my go at doing this

https://github.com/steveloughran/formality/blob/master/styleguide/styleguide.md


I've not done any work on trying to get it in, more evolving it as how I code & 
what I look for, especially in tests.

If you want to take this on, it'd be nice. At the same time, I fear there'd be 
push back if you turned up and started telling people what to do. Collaborating 
with us all on the test code is a good place to start.

We're also more relaxed about contributions to the less-core bits of the system 
(things like HDFS, IPC, security and Yarn core are trouble). If there's stuff 
outside that you want to take a go at helping clean up, that'd be lower risk 
(example: object store connectors)

-Steve



On 7 Aug 2017, at 13:13, Lars Francke 
> wrote:

Hi,

a few words about me: I've contributed to Hadoop (and it's ecosystem[4]) in
the past am a Hive committer and have used Hadoop for 10 years now, so I'm
not totally inexperienced. I'm earning my money as a Hadoop consultant so
I've seen dozens of real-life clusters in my life.

As part of a few recent client projects and now writing about Hadoop in a
new project/book I'm digging into the source code to figure out some of the
things that are not documented.

But as part of this digging I'm seeing lots of warnings in the code,
inconsistencies etc. and I'd like to contribute some fixes to this back to
the community.

I have been a long-time believer in good code quality and consistent code
styles. This might affect people like me especially who do a lot of
"drive-by" contributions as I'm not someone who looks at the code daily but
comes across it reasonably often as part of client engagements. In those
scenarios, it's very unhelpful to have inconsistent code & bad
documentation.

Two simple but concrete examples:
* There's lots of "final" usages on variables and methods but no
consistency. Was this done for particular reasons or personal preference?

personal, though with a move to l-expressions, it matters a lot more. We should 
really be marking all parameters as final at the very least.


* Similarly, there's lots of things that are public or protected while they
could in theory be private. This especially makes it very hard to reason
about code.

there's now a bit of fear of breaking things, but at the very least, things 
could be protected or package-private more than they are.



Judging from the current code there's lots of "unofficial" code styling
and/or personal preference. The Wiki says[1] to follow the Sun
guidelines[2] which have not been updated in almost 20 years. A new version
is in the works an clarifies a lot of things[3]. I'm trying to get it
published soon. I'd try to format according to the latter (that means among
other things no "final" for local variables).

I realize that I won't be able to single-handedly fix all of this
especially as code gets contributed but if the community thinks it's
worthwhile I'd still love to land a few cleanup patches. My experience in
the past has been that it's hard to get attention to these things (which I
fully understand as they take up someone's time to review & commit).

So, this is my request for comments on these questions:
* Is there any interest in this at all?
** "This" being patches for code style & things like FindBugs & Checkstyle
warnings
* Size of the patches: Rather one big patch or smaller ones (e.g. per file
or package)
* Anyone willing to help me with this? e.g. reviewing and committing? I'd
be more than happy to bribe you with drinks, sweets, food or something else

My plan is not to go through each and every file and fix every issue I see.
But there are some specific areas I'm looking at in detail and there I'd
love to contribute back.

Thank you for reading!

Cheers,
Lars

PS: Posting to common-dev only, not sure if I should cross post to hdfs-dev
and yarn-dev as well?

[1] 
[2] <
http://www.oracle.com/technetwork/java/javase/documentation/codeconvtoc-136057.html

[3] 
[4] <
https://issues.apache.org/jira/issues/?filter=-1=reporter%20%3D%20lars_francke%20OR%20assignee%20%3D%20lars_francke

Question about how to best contribute

[Lars Francke]

Hi,

a few words about me: I've contributed to Hadoop (and it's ecosystem[4]) in
the past am a Hive committer and have used Hadoop for 10 years now, so I'm
not totally inexperienced. I'm earning my money as a Hadoop consultant so
I've seen dozens of real-life clusters in my life.

As part of a few recent client projects and now writing about Hadoop in a
new project/book I'm digging into the source code to figure out some of the
things that are not documented.

But as part of this digging I'm seeing lots of warnings in the code,
inconsistencies etc. and I'd like to contribute some fixes to this back to
the community.

I have been a long-time believer in good code quality and consistent code
styles. This might affect people like me especially who do a lot of
"drive-by" contributions as I'm not someone who looks at the code daily but
comes across it reasonably often as part of client engagements. In those
scenarios, it's very unhelpful to have inconsistent code & bad
documentation.

Two simple but concrete examples:
* There's lots of "final" usages on variables and methods but no
consistency. Was this done for particular reasons or personal preference?
* Similarly, there's lots of things that are public or protected while they
could in theory be private. This especially makes it very hard to reason
about code.

Judging from the current code there's lots of "unofficial" code styling
and/or personal preference. The Wiki says[1] to follow the Sun
guidelines[2] which have not been updated in almost 20 years. A new version
is in the works an clarifies a lot of things[3]. I'm trying to get it
published soon. I'd try to format according to the latter (that means among
other things no "final" for local variables).

I realize that I won't be able to single-handedly fix all of this
especially as code gets contributed but if the community thinks it's
worthwhile I'd still love to land a few cleanup patches. My experience in
the past has been that it's hard to get attention to these things (which I
fully understand as they take up someone's time to review & commit).

So, this is my request for comments on these questions:
* Is there any interest in this at all?
** "This" being patches for code style & things like FindBugs & Checkstyle
warnings
* Size of the patches: Rather one big patch or smaller ones (e.g. per file
or package)
* Anyone willing to help me with this? e.g. reviewing and committing? I'd
be more than happy to bribe you with drinks, sweets, food or something else

My plan is not to go through each and every file and fix every issue I see.
But there are some specific areas I'm looking at in detail and there I'd
love to contribute back.

Thank you for reading!

Cheers,
Lars

PS: Posting to common-dev only, not sure if I should cross post to hdfs-dev
and yarn-dev as well?

[1] 
[2] <
http://www.oracle.com/technetwork/java/javase/documentation/codeconvtoc-136057.html
>
[3] 
[4] <
https://issues.apache.org/jira/issues/?filter=-1=reporter%20%3D%20lars_francke%20OR%20assignee%20%3D%20lars_francke
>

Re: HADOOP-14163 proposal for new hadoop.apache.org

[Akira Ajisaka]

Hi folks,

I'd like to create a separate git repository for the web site (now managed in 
svn).
For more details, please see the discussion in 
https://issues.apache.org/jira/browse/HADOOP-14163.

I'll ask INFRA to create a git repository if there are no objections.

Thanks Marton Elek for the continuous work.

Regards,
Akira

On 2017/03/25 3:35, Andrew Wang wrote:

Thanks again for working on this Marton!

Based on my read of the blog post you linked, we should have the git branch
ready before asking infra to switch it over.

I can do a more detailed review on the JIRA once you rev, and can help with
the INFRA ticket once it's ready. We'll also have to update BUILDING.txt
and the wiki instructions as part of this.

Best,
Andrew

On Fri, Mar 24, 2017 at 3:06 AM, Marton Elek  wrote:




Thank you all of the feedbacks, I fixed all of them (except one, see the
comment below) and updated the http://hadoop.anzix.net preview site.

So the next steps:

0. Let me know if you have any comment about the latest version

1. I wait for the 2.8.0 announcement, and migrate the new announcement as
well. (wouldn't like to complicate the 2.8.0 with the site change)

2. I like the suggestion of Owen to move the site to a specific git
branch. I wouldn't like to pending on it if it's too much time, but if any
of the commiters could pick it up, I would wait for it.

I tested it, and seems to be easy:

git svn clone https://svn.apache.org/repos/asf/hadoop/common/site/main
cd main
git remote add elek g...@github.com:elek/hadoop.git
git push elek master:asf-site

According to the blog entry, an INFRA issue should be opened (I guess by a
commiter or maybe a pmc member):

https://blogs.apache.org/infra/entry/git_based_websites_available

3. After that I can submit the new site as a regular patch against the
asf-site branch.

4. If it's merged, I can update the release wiki pages

Marton

ps:

The only suggested item which is not implemented is the short version
names in the documentation menu (2.7 instead of 2.7.3).

I think there are two forces: usability of the site and the simplicity of
the site generation. Ideally a new release could be added to the site as
easy as possible (that was one of the motivation of the migration).

While a new tag could be added to the header of the markdown files (eg:
versionLine: 3.0), it requires multiple files update during a new release.
And if something would be missed, there could be displayed multiple "2.7"
menu item (one for 2.7.3 and for 2.7.4). So the current method is not so
nice, but much more bug-safe.

I prefer to keep the current/content in this step (if possible) and if the
site is migrated we can submit new patches (hopefully against a git branch)
in the normal way and further improve the site.



From: Owen O'Malley 
Sent: Monday, March 13, 2017 6:15 PM
To: Marton Elek
Cc: common-dev@hadoop.apache.org
Subject: Re: HADOOP-14163 proposal for new hadoop.apache.org

Thanks for addressing this. Getting rid of Hadoop's use of forrest is a
good thing.

In terms of content, the documentation links should be sorted by number
with only the latest from each minor release line (eg. 3.0, 2.7, 2.6).

The download page points to the mirrors for checksums and signatures. It
should use the direct links, such as

https://dist.apache.org/repos/dist/release/hadoop/common/
hadoop-2.7.3/hadoop-2.7.3-src.tar.gz.asc
https://dist.apache.org/repos/dist/release/hadoop/common/
hadoop-2.7.3/hadoop-2.7.3-src.tar.gz.mds

Speaking of which, Hadoop's dist directory is huge and should be heavily
pruned. We should probably take it down to just hadoop-2.6.5, hadoop-2.7.3,
and hadoop-3.0.0-alpha2.

You might also want to move us to git-pubsub so that we can use a branch in
our source code git repository to publish the html. Typically this uses the
asf-site branch.

.. Owen

On Mon, Mar 13, 2017 at 7:28 AM, Marton Elek 
wrote:



Hi,

In the previous thread the current forrest based hadoop site is

identified

as one of the pain points of the release process.

I created a new version of the site with exactly the same content.

 As it uses newer site generator (hugo), now:

1. It’s enough to create one new markdown file per release, and all the
documentation/download links will be automatically added.
2. It requires only one single binary to render.


A preview version is temporary hosted at

 http://hadoop.anzix.net/

to make it easier to review.


For more details, you can check my comments on the issue
https://issues.apache.org/jira/browse/HADOOP-14163

I would be thankful to get any feedback/review.

Cheers,
Marton





-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org






-
To unsubscribe, 

[jira] [Created] (HADOOP-14739) Add build instruction for docker on Mac instead of docker toolbox.

[Akira Ajisaka (JIRA)]

Akira Ajisaka created HADOOP-14739:
--

 Summary: Add build instruction for docker on Mac instead of docker 
toolbox.
 Key: HADOOP-14739
 URL: https://issues.apache.org/jira/browse/HADOOP-14739
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Reporter: Akira Ajisaka


HADOOP-12575 added build instruction for docker toolbox.
Now Docker for Mac (https://www.docker.com/docker-mac) is available and it can 
skip some procedures written in BUILDING.txt.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.2 Release Plan

[Junping Du]

Hello community,
Here is a quick update on status for 2.8.2:
- We are 0 blockers now!
- Still 9 critical issues, 8 of them are Patch Available and with actively 
working.
For details of pending blocker/critical issues, please refer: 
https://s.apache.org/JM5x

I am planning to cut off first RC in week of Aug. 21st to give these 
critical issues a bit more time (~2 weeks) to get fixed. Let's working towards 
first production GA release of Apache Hadoop 2.8 - let me know if you have any 
thoughts or comments.

Cheers,

Junping

From: Junping Du 
Sent: Monday, July 24, 2017 1:41 PM
To: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re:

I have done the change.

All committers,

  2.8.2 release is supposed to be a stable/production release for 
branch-2.8. For commits to go for 2.8.2 release (only important and low risk 
bug fixes), please commit to trunk, branch-2, branch-2.8 and branch-2.8.2. For 
unimportant or high risk bug fixes/improvements, please commit to branch-2.8 
(trunk/branch-2) only and mark JIRA fixed as 2.8.3. Thanks for your cooperation!


Thanks,


Junping



From: Junping Du
Sent: Monday, July 24, 2017 10:36 AM
To: Brahma Reddy Battula; Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan


Nice catch, Brahma. Actually, this is not supposed to be happen as we all 
should know the patch should firstly get landed on branch-2.8 (as well as 
trunk, branch-2) before landed on branch-2.8.2. Anyway, we will always check 
JIRAs claim to fixed in a specific release version with commits actually 
landing on the releasing branch before kicking off RC. So, I am not too worry 
about this mistaking behaviors as it happens in every releases.

If no other concerns, I will do the branch update in next 30 minutes.


Thanks,


Junping



From: Brahma Reddy Battula 
Sent: Sunday, July 23, 2017 1:50 AM
To: Junping Du; Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan



Just executed the "git log branch-2.8 ^branch-2.8.2" found two commits are 
missed (HDFS-8312 and HADOOP-13867 ) in branch-2.8.I just pushed this two 
commits.Hope we'll not miss any commits which present in only in branch-2.8.2.


From: Junping Du 
Sent: Saturday, July 22, 2017 5:57 AM
To: Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Already get back from Daniel who is from ASF INFRA team, I plan to do following 
operations on next Monday morning:
1. Drop current branch-2.8.2 and recut branch-2.8.2 from branch-2.8
2. Drop abandoned branch-2.8.1 and rename branch-2.8.1-private to branch-2.8.1 
where we just released 2.8.1 from.
I will also adjust fix version on all affected JIRA accordingly.

If you have any concerns on above operations, please raise it before the end of 
this Sunday (7/23).


Thanks,

Junping


From: Junping Du 
Sent: Friday, July 21, 2017 2:29 PM
To: Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Make sense, just raise: https://issues.apache.org/jira/browse/INFRA-14669

Thanks,

Junping

From: Vinod Kumar Vavilapalli 
Sent: Friday, July 21, 2017 12:31 PM
To: Junping Du
Cc: Kihwal Lee; common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Junping,

If we are looking at a month, I’d not rebranch branch-2.8.2 right now given how 
these things go. We can just continue to commit on branch-2.8 for now.

I also think we should just follow up with ASF INFRA and clean up the branches
 - Delete branch-2.8.2 so that we can recreate it afresh a little later.
 - branch-2.8.1 is also stale and it should be deleted. branch-2.8.1-private 
should be renamed to branch-2.8.1

Thanks
+Vinod

> On Jul 21, 2017, at 11:23 AM, Junping Du  wrote:
>
> Thanks for suggestions, Jason and Kihwal!
> +1 on releasing 2.8.2 on latest branch-2.8 too.