date:20170908

Re: Permissions to edit Confluence Wiki

2017-09-08 Thread Arun Suresh

Thanks Akira

Cheers
-Arun

On Fri, Sep 8, 2017 at 6:53 PM, Akira Ajisaka  wrote:

> You can get the privilege by sending an e-mail to the dev ML.
> I added it for you.
>
> Thanks,
> Akira
>
>
> On 2017/09/09 4:50, Arun Suresh wrote:
>
>> Hi folks
>>
>> How do we get access to edit the confluence wiki;
>> https://cwiki.apache.org/confluence/display/HADOOP ?
>>
>> We were hoping to update it with hadoop 2.9 release details.
>>
>> Cheers
>> -Arun
>>
>>

Re: Permissions to edit Confluence Wiki

2017-09-08 Thread Akira Ajisaka


You can get the privilege by sending an e-mail to the dev ML.
I added it for you.

Thanks,
Akira

On 2017/09/09 4:50, Arun Suresh wrote:

Hi folks

How do we get access to edit the confluence wiki;
https://cwiki.apache.org/confluence/display/HADOOP ?

We were hoping to update it with hadoop 2.9 release details.

Cheers
-Arun



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] Merge yarn-native-services branch into trunk

2017-09-08 Thread Jian He

Hi Arun

Sorry for late reply.
* Is there a branch-2 merge planned for this ?
Branch-2 is not planned for this merge.

* I understand YARN-7126 has some introductory documentation, But I think we 
need to flesh it up a bit more before release, I would also like to see steps 
to deploy a sample service.
We have added more documentations, QuickStart.md, Overview.md and others in the 
same folder.
YarnCommands.md is also updated to document the new shell commands.

I encourage everyone to try  and share suggestions.

As said in another email thread, we decided to drop this for beta and re-target 
it for GA.

Thanks,
Jian

On Sep 5, 2017, at 1:37 PM, Arun Suresh 
> wrote:

Thanks for all the work on this folks.
I know the VOTE thread has started for this.

But I did have a couple of questions:
* Is there a branch-2 merge planned for this ?
* I understand YARN-7126 has some introductory documentation, But I think we 
need to flesh it up a bit more before release, I would also like to see steps 
to deploy a sample service.

Cheers
-Arun

On Thu, Aug 31, 2017 at 12:40 AM, Jian He 
> wrote:
Update:
I’ve chatted with Andrew offline, we’ll proceed with merging 
yarn-native-services into trunk for beta.
We’ll advertise this feature as “alpha"
Currently, we have completed all the jiras for this merge - I’ve also moved out 
the subtasks that are not blocking this merge.

I’ve created YARN-7127 to run the entire patch against trunk, once that goes 
green, I plan to start a formal vote.

Thanks,
Jian

On Aug 18, 2017, at 2:48 PM, Andrew Wang 
>>
 wrote:

Hi Jian, thanks for the reply,

On Thu, Aug 17, 2017 at 1:03 PM, Jian He 
>>
 wrote:
Thanks Andrew for the comments. Answers below:

- There are no new APIs added in YARN/Hadoop core. In fact, all the new code 
are running outside of existing system and they are optional and require users 
to explicitly opt in. The new system’s own rest API is not stable and will be 
evolving.

Great! That adds a lot more confidence that this is safe to merge.

Are these new APIs listed in user documentation, and described as unstable?

- We have been running/testing a version of the entire system internally for 
quite a while.

Do you mind elaborating on the level of testing? Number of nodes, types of 
applications, production or test workload, etc. It'd help us build confidence.

- I’d like to see this in hadoop3-beta1. Of course, we’ll take responsibility 
of moving fast and not block the potential timeline.

Few more questions:

How should we advertise this feature in the release? Since the APIs are 
unstable, I'd propose calling it "alpha" in the release notes, like we do the 
TSv2.

Could you move out subtasks from YARN-5079 that are not blocking the merge? 
This would make it easier to understand what's remaining.

Thanks,
Andrew

Re: [VOTE] Merge yarn-native-services branch into trunk

2017-09-08 Thread Jian He

Hi Andrew,

At this point, there are no more release blockers including documentations from 
our side - all work done.
But I agree it is too close to the release, after talking with other team 
members, we are fine to drop  this from beta,

And we want to target this for GA.
I’m withdrawing this vote and will start afresh vote later for GA. 
Thanks all who voted this effort !

Thanks,
Jian


> On Sep 7, 2017, at 3:59 PM, Andrew Wang  wrote:
> 
> Hi folks,
> 
> This vote closes today. I see a -1 from Allen on inclusion in beta1. I see
> there's active fixing going on, but given that we're one week out from RC0,
> I think we should drop this from beta1.
> 
> Allen, Jian, others, is this reasonable? What release should we retarget
> this for? I don't have a sense for how much work there is left to do, but
> as a reminder, we're planning GA for Nov 1st, and 3.1.0 for January.
> 
> Best,
> Andrew
> 
> On Wed, Sep 6, 2017 at 10:19 AM, Jian He  wrote:
> 
>>>  Please correct me if I’m wrong, but the current summary of the
>> branch, post these changes, looks like:
>> Sorry for confusion, I was actively writing the formal documentation for
>> how to use/how it works etc. and will post soon in a few hours.
>> 
>> 
>>> On Sep 6, 2017, at 10:15 AM, Allen Wittenauer 
>> wrote:
>>> 
>>> 
 On Sep 5, 2017, at 6:23 PM, Jian He  wrote:
 
>If it doesn’t have all the bells and whistles, then it shouldn’t
>> be on port 53 by default.
 Sure, I’ll change the default port to not use 53 and document it.
>*how* is it getting launched on a privileged port? It sounds like
>> the expectation is to run “command” as root.   *ALL* of the previous
>> daemons in Hadoop that needed a privileged port used jsvc.  Why isn’t this
>> one? These questions matter from a security standpoint.
 Yes, it is running as “root” to be able to use the privileged port. The
>> DNS server is not yet integrated with the hadoop script.
 
> Check the output.  It’s pretty obviously borked:
 Thanks for pointing out. Missed this when rebasing onto trunk.
>>> 
>>> 
>>>  Please correct me if I’m wrong, but the current summary of the
>> branch, post these changes, looks like:
>>> 
>>>  * A bunch of mostly new Java code that may or may not have
>> javadocs (post-revert YARN-6877, still working out HADOOP-14835)
>>>  * ~1/3 of the docs are roadmap/TBD
>>>  * ~1/3 of the docs are for an optional DNS daemon that has
>> no end user hook to start it
>>>  * ~1/3 of the docs are for a REST API that comes from some
>> undefined daemon (apiserver?)
>>>  * Two new, but undocumented, subcommands to yarn
>>>  * There are no docs for admins or users on how to actually
>> start or use this completely new/separate/optional feature
>>> 
>>>  How are outside people (e.g., non-branch committers) supposed to
>> test this new feature under these conditions?
>>> 
>> 
>> 
>> -
>> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>> 
>>

Permissions to edit Confluence Wiki

2017-09-08 Thread Arun Suresh

Hi folks

How do we get access to edit the confluence wiki;
https://cwiki.apache.org/confluence/display/HADOOP ?

We were hoping to update it with hadoop 2.9 release details.

Cheers
-Arun

[jira] [Resolved] (HADOOP-14650) Update jsp-api version

2017-09-08 Thread Ray Chiang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-14650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ray Chiang resolved HADOOP-14650.
-
Resolution: Won't Fix
  Assignee: Ray Chiang

> Update jsp-api version
> --
>
> Key: HADOOP-14650
> URL: https://issues.apache.org/jira/browse/HADOOP-14650
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>
> The current artifact is:
> javax.servlet.jsp:jsp-api:2.1
> That version could either be bumped to 2.1.2 (the latest of that line), or 
> use the latest artifact:
> javax.servlet.jsp:javax.servlet.jsp-api:2.3.1



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14656) Update xercesImpl version to 2.11.0

2017-09-08 Thread Ray Chiang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-14656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ray Chiang resolved HADOOP-14656.
-
Resolution: Won't Fix
  Assignee: Ray Chiang

> Update xercesImpl version to 2.11.0
> ---
>
> Key: HADOOP-14656
> URL: https://issues.apache.org/jira/browse/HADOOP-14656
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: 3.0.0-beta1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>
> Update the dependency
> xerces:xercesImpl:2.9.1
> to the latest (2.11.0).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14855) Hadoop scripts may errantly believe a daemon is still running, preventing it from starting

2017-09-08 Thread Aaron T. Myers (JIRA)

Aaron T. Myers created HADOOP-14855:
---

 Summary: Hadoop scripts may errantly believe a daemon is still 
running, preventing it from starting
 Key: HADOOP-14855
 URL: https://issues.apache.org/jira/browse/HADOOP-14855
 Project: Hadoop Common
  Issue Type: Bug
  Components: scripts
Affects Versions: 3.0.0-alpha4
Reporter: Aaron T. Myers


I encountered a case recently where the NN wouldn't start, with the error 
message "namenode is running as process 16769.  Stop it first." In fact the NN 
was not running at all, but rather another long-running process was running 
with this pid.

It looks to me like our scripts just check to see if _any_ process is running 
with the pid that the NN (or any Hadoop daemon) most recently ran with. This is 
clearly not a fool-proof way of checking to see if a particular type of daemon 
is now running, as some other process could start running with the same pid 
since the daemon in question was previously shut down.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14854) DistCp should not issue file status calls for files in the filter list

2017-09-08 Thread Mukul Kumar Singh (JIRA)

Mukul Kumar Singh created HADOOP-14854:
--

 Summary: DistCp should not issue file status calls for files in 
the filter list
 Key: HADOOP-14854
 URL: https://issues.apache.org/jira/browse/HADOOP-14854
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Mukul Kumar Singh
Assignee: Mukul Kumar Singh


DistCp currently excludes the files in the filter list only when the files are 
added to the copy list.
However distcp can be optimized by not issuing file status/get attr calls for 
the files in the filter.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-09-08 Thread Apache Jenkins Server

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/517/

[Sep 7, 2017 7:46:20 AM] (sunilg) YARN-6992. Kill application button is visible 
even if the application is
[Sep 7, 2017 12:38:23 PM] (kai.zheng) HDFS-12402. Refactor 
ErasureCodingPolicyManager and related codes.
[Sep 7, 2017 3:18:28 PM] (arp) HDFS-12376. Enable JournalNode Sync by default. 
Contributed by Hanisha
[Sep 7, 2017 4:50:36 PM] (yzhang) HDFS-12357. Let NameNode to bypass external 
attribute provider for
[Sep 7, 2017 5:35:03 PM] (stevel) HADOOP-14520. WASB: Block compaction for 
Azure Block Blobs. Contributed
[Sep 7, 2017 5:23:12 PM] (Arun Suresh) YARN-6978. Add updateContainer API to 
NMClient. (Kartheek Muthyala via
[Sep 7, 2017 6:55:56 PM] (stevel) HADOOP-14774. S3A case 
"testRandomReadOverBuffer" failed due to improper
[Sep 7, 2017 7:40:09 PM] (aengineer) HDFS-12350. Support meta tags in configs. 
Contributed by Ajay Kumar.
[Sep 7, 2017 9:13:37 PM] (wangda) YARN-7033. Add support for NM Recovery of 
assigned resources (e.g.
[Sep 7, 2017 9:17:03 PM] (jlowe) YARN-6930. Admins should be able to explicitly 
enable specific
[Sep 7, 2017 11:30:12 PM] (xiao) HDFS-12369. Edit log corruption due to hard 
lease recovery of not-closed
[Sep 7, 2017 11:56:35 PM] (wang) HDFS-12218. Rename split EC / replicated block 
metrics in BlockManager.
[Sep 7, 2017 11:57:19 PM] (wang) HDFS-12218. Addendum. Rename split EC / 
replicated block metrics in
[Sep 8, 2017 12:20:42 AM] (manojpec) HDFS-12404. Rename hdfs config 
authorization.provider.bypass.users to
[Sep 8, 2017 1:01:37 AM] (lei) HDFS-12349. Improve log message when it could 
not alloc enough blocks
[Sep 8, 2017 1:45:17 AM] (sunilg) YARN-6600. Introduce default and max lifetime 
of application at
[Sep 8, 2017 2:07:17 AM] (subru) YARN-5330. SharingPolicy enhancements required 
to support recurring
[Sep 8, 2017 3:51:02 AM] (xiao) HDFS-12400. Provide a way for NN to drain the 
local key cache before




-1 overall


The following subsystems voted -1:
findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-hdfs-project/hadoop-hdfs 
   Format-string method String.format(String, Object[]) called with format 
string "File %s could only be written to %d of the %d %s. There are %d 
datanode(s) running and %s node(s) are excluded in this operation." wants 6 
arguments but is given 7 in 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(String,
 int, Node, Set, long, List, byte, BlockType, ErasureCodingPolicy, EnumSet) At 
BlockManager.java:with format string "File %s could only be written to %d of 
the %d %s. There are %d datanode(s) running and %s node(s) are excluded in this 
operation." wants 6 arguments but is given 7 in 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(String,
 int, Node, Set, long, List, byte, BlockType, ErasureCodingPolicy, EnumSet) At 
BlockManager.java:[line 2076] 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 
   Hard coded reference to an absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:[line 490] 

Failed junit tests :

   hadoop.hdfs.server.namenode.TestReencryption 
   hadoop.hdfs.server.namenode.TestReencryptionWithKMS 
   hadoop.hdfs.TestLeaseRecoveryStriped 
   hadoop.hdfs.TestClientProtocolForPipelineRecovery 
   hadoop.hdfs.TestReconstructStripedFile 
   hadoop.hdfs.server.blockmanagement.TestBlockManager 
   hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean 
   hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation 
   hadoop.yarn.server.TestDiskFailures 
   hadoop.mapreduce.v2.hs.webapp.TestHSWebApp 
   hadoop.yarn.sls.TestReservationSystemInvariants 
   hadoop.yarn.sls.TestSLSRunner 

Timed out junit tests :

   org.apache.hadoop.hdfs.TestWriteReadStripedFile 
   
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA 
   
org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/517/artifact/out/diff-compile-cc-root.txt
  [4.0K]

[jira] [Created] (HADOOP-14853) hadoop-mapreduce-client-app is not a client module

2017-09-08 Thread Haibo Chen (JIRA)

Haibo Chen created HADOOP-14853:
---

 Summary: hadoop-mapreduce-client-app is not a client module
 Key: HADOOP-14853
 URL: https://issues.apache.org/jira/browse/HADOOP-14853
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0-alpha4
Reporter: Haibo Chen
Assignee: Haibo Chen


hadoop-mapreduce-client-app is not a client module, and thus can be removed as 
a dependency from hadoop-client module



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Merge yarn-native-services branch into trunk

2017-09-08 Thread Jian He

Hi Allen,
The documentations are committed. Please check QuickStart.md and others in the 
same folder.
YarnCommands.md doc is updated to include new commands.
DNS default port is also documented. 
Would you like to give a look and see if it address your concerns ?

Jian

> On Sep 6, 2017, at 10:19 AM, Jian He  wrote:
> 
>>  Please correct me if I’m wrong, but the current summary of the branch, 
>> post these changes, looks like:
> Sorry for confusion, I was actively writing the formal documentation for how 
> to use/how it works etc. and will post soon in a few hours.
> 
> 
>> On Sep 6, 2017, at 10:15 AM, Allen Wittenauer  
>> wrote:
>> 
>> 
>>> On Sep 5, 2017, at 6:23 PM, Jian He  wrote:
>>> 
If it doesn’t have all the bells and whistles, then it shouldn’t be on 
 port 53 by default.
>>> Sure, I’ll change the default port to not use 53 and document it.
*how* is it getting launched on a privileged port? It sounds like the 
 expectation is to run “command” as root.   *ALL* of the previous daemons 
 in Hadoop that needed a privileged port used jsvc.  Why isn’t this one? 
 These questions matter from a security standpoint.  
>>> Yes, it is running as “root” to be able to use the privileged port. The DNS 
>>> server is not yet integrated with the hadoop script. 
>>> 
 Check the output.  It’s pretty obviously borked:
>>> Thanks for pointing out. Missed this when rebasing onto trunk.
>> 
>> 
>>  Please correct me if I’m wrong, but the current summary of the branch, 
>> post these changes, looks like:
>> 
>>  * A bunch of mostly new Java code that may or may not have 
>> javadocs (post-revert YARN-6877, still working out HADOOP-14835)
>>  * ~1/3 of the docs are roadmap/TBD
>>  * ~1/3 of the docs are for an optional DNS daemon that has no 
>> end user hook to start it
>>  * ~1/3 of the docs are for a REST API that comes from some 
>> undefined daemon (apiserver?)
>>  * Two new, but undocumented, subcommands to yarn
>>  * There are no docs for admins or users on how to actually 
>> start or use this completely new/separate/optional feature
>> 
>>  How are outside people (e.g., non-branch committers) supposed to test 
>> this new feature under these conditions?
>> 
>

[jira] [Created] (HADOOP-14852) Intermittent failure of S3Guard TestConsistencyListFiles

2017-09-08 Thread Steve Loughran (JIRA)

Steve Loughran created HADOOP-14852:
---

 Summary: Intermittent failure of S3Guard TestConsistencyListFiles
 Key: HADOOP-14852
 URL: https://issues.apache.org/jira/browse/HADOOP-14852
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3, test
Affects Versions: 3.0.0-beta1
Reporter: Steve Loughran


I'm seeing intermittent test failures with a test run of {{ -Dparallel-tests 
-DtestsThreadCount=8 -Ds3guard -Ddynamo}}  (-Dauth set or unset) in which a 
file in DELAY-LISTING-ME isn't being returned in a listing. 

Theories
* test is wrong
* config is wrong
* code is wrong



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-9383) mvn clean compile fails without install goal

2017-09-08 Thread Eric Payne (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Payne reopened HADOOP-9383:


Something similar is definitely still happening on fedora and redhat. This is 
what I'm getting.
{noformat}
Could not find artifact 
org.apache.hadoop:hadoop-maven-plugins:jar:3.1.0-SNAPSHOT
{noformat}
I'm reopening the JIRA.

> mvn clean compile fails without install goal
> 
>
> Key: HADOOP-9383
> URL: https://issues.apache.org/jira/browse/HADOOP-9383
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Arpit Agarwal
>
> 'mvn -Pnative-win clean compile' fails with the following error:
> [ERROR] Could not find goal 'protoc' in plugin 
> org.apache.hadoop:hadoop-maven-plugins:3.0.0-SNAPSHOT among available goals 
> -> [Help 1]
> The build succeeds if the install goal is specified.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-8391) Hadoop-auth should use log4j

2017-09-08 Thread Andras Bokor (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Bokor resolved HADOOP-8391.
--
Resolution: Won't Fix

I agree with Steve. Hadoop modules are moving toward SLF4J.

> Hadoop-auth should use log4j
> 
>
> Key: HADOOP-8391
> URL: https://issues.apache.org/jira/browse/HADOOP-8391
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: conf
>Affects Versions: 2.0.0-alpha
>Reporter: Eli Collins
>
> Per HADOOP-8086 hadoop-auth uses slf4j, don't see why it shouldn't use log4j 
> to be consistent with the rest of Hadoop.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[DISCUSS] official docker image(s) for hadoop

2017-09-08 Thread Marton, Elek



TL;DR: I propose to create official hadoop images and upload them to the 
dockerhub.


GOAL/SCOPE: I would like improve the existing documentation with 
easy-to-use docker based recipes to start hadoop clusters with various 
configuration.


The images also could be used to test experimental features. For example 
ozone could be tested easily with these compose file and configuration:


https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6

Or even the configuration could be included in the compose file:

https://github.com/elek/hadoop/blob/docker-2.8.0/example/docker-compose.yaml

I would like to create separated example compose files for federation, 
ha, metrics usage, etc. to make it easier to try out and understand the 
features.


CONTEXT: There is an existing Jira 
https://issues.apache.org/jira/browse/HADOOP-13397
But it’s about a tool to generate production quality docker images 
(multiple types, in a flexible way). If no objections, I will create a 
separated issue to create simplified docker images for rapid prototyping 
and investigating new features. And register the branch to the dockerhub 
to create the images automatically.


MY BACKGROUND: I am working with docker based hadoop/spark clusters 
quite a while and run them succesfully in different environments 
(kubernetes, docker-swarm, nomad-based scheduling, etc.) My work is 
available from here: https://github.com/flokkr but they could handle 
more complex use cases (eg. instrumenting java processes with btrace, or 
read/reload configuration from consul).
 And IMHO in the official hadoop documentation it’s better to suggest 
to use official apache docker images and not external ones (which could 
be changed).


Please let me know if you have any comments.

Marton

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-9383) mvn clean compile fails without install goal

2017-09-08 Thread Andras Bokor (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Bokor resolved HADOOP-9383.
--
Resolution: Cannot Reproduce

> mvn clean compile fails without install goal
> 
>
> Key: HADOOP-9383
> URL: https://issues.apache.org/jira/browse/HADOOP-9383
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Arpit Agarwal
>
> 'mvn -Pnative-win clean compile' fails with the following error:
> [ERROR] Could not find goal 'protoc' in plugin 
> org.apache.hadoop:hadoop-maven-plugins:3.0.0-SNAPSHOT among available goals 
> -> [Help 1]
> The build succeeds if the install goal is specified.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

hadoop roadmaps

2017-09-08 Thread Marton, Elek


Hi,

I tried to summarize all of the information from different mail threads 
about the upcomming releases:


https://cwiki.apache.org/confluence/display/HADOOP/Roadmap

Please fix it / let me know if you see any invalid data. I will try to 
follow the conversations and update accordingly.


Two administrative questions:

 * Is there any information about which wiki should be used? Or about 
the migration process? As I see the new pages are created on the cwiki 
recently.


 * Could you please give me permission (user: elek) to the old wiki. I 
would like to update the old Roadmap page 
(https://wiki.apache.org/hadoop/Roadmap)


Thanks
Marton

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14851) LambdaTestUtils.eventually() doesn't spin on Assertion failures

2017-09-08 Thread Steve Loughran (JIRA)

Steve Loughran created HADOOP-14851:
---

 Summary: LambdaTestUtils.eventually() doesn't spin on Assertion 
failures
 Key: HADOOP-14851
 URL: https://issues.apache.org/jira/browse/HADOOP-14851
 Project: Hadoop Common
  Issue Type: Bug
  Components: test
Affects Versions: 2.8.1
Reporter: Steve Loughran
Assignee: Steve Loughran


This is funny. The {{LsmbdaTestUtils.eventually()}} method, meant to spin until 
a closure stops raising exceptions, doesn't catch {{Error}} and subclasses, so 
doesn't fail on an {{Assert.assert()}} failure, which raises an 
{{AssertionError}}. My bad :)

Example:
{code}
eventually(TIMEOUT,
() -> {
  while (counter.incrementAndGet()  < 5) {
assert false : "oops";
  }
},
retryLogic);
{code}

Fix: catch Throwable, rethrow. Needs to add VirtualMachineError & subclasses to 
the set of errors not to spin on (OOM, stack overflow, ...)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14850) Read HttpServer2 resources directly from the source tree (if exists)

2017-09-08 Thread Elek, Marton (JIRA)

Elek, Marton created HADOOP-14850:
-

 Summary: Read HttpServer2 resources directly from the source tree 
(if exists)
 Key: HADOOP-14850
 URL: https://issues.apache.org/jira/browse/HADOOP-14850
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 3.0.0-alpha4
Reporter: Elek, Marton
Assignee: Elek, Marton


Currently the Hadoop server components can't be started from IDE during the 
development. There are two reasons for that:

1. some artifacts are in provided scope which are definitelly needed to run the 
server (see HDFS-12197)

2. The src/main/webapp dir should be on the classpath (but not).

In this issue I suggest to fix the second issue by reading the web resources 
(html and css files) directly from the source tree and not from the classpath 
but ONLY if the src/main/webapp dir exists. Similar approach exists in 
different projects (eg. in Spark).

WIth this patch the web development of the web interfaces are significant 
easier as the result could be checked immediatelly with a running severt 
(without rebuild/restart). I used this patch during the development of the 
Ozone web interfaces.

As the original behaviour of the resource location has not been change if 
"src/main/webapp" doesn't exist, I think it's quite safe.  And the method is 
called only once during the creation of the HttpServer2 there is also no change 
in performance.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: 2017-09-07 Hadoop 3 release status update

2017-09-08 Thread Steve Loughran


On 8 Sep 2017, at 00:50, Andrew Wang 
> wrote:

  - HADOOP-14738  (Remove
  S3N and obsolete bits of S3A; rework docs): Steve has been actively revving
  this with our new committer Aaron Fabbri ready to review. The scope has
  expanded from HADOOP-14826, so it's not just a doc update.

For people not tracking this, it's merged with other cleanup code so pulls the 
entirety of the s3n:// connector and the original 
S3AOutputStreamessentially the unmaintained and obsolete bits of code. The 
ones where any bugrep would be dealt with "have you switched to..."

[jira] [Created] (HADOOP-14849) some wrong spelling words update

2017-09-08 Thread Chen Hongfei (JIRA)

Chen Hongfei created HADOOP-14849:
-

 Summary: some wrong spelling words update
 Key: HADOOP-14849
 URL: https://issues.apache.org/jira/browse/HADOOP-14849
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Chen Hongfei
Priority: Trivial


Wrong spelling "refered" should be updated to "referred";
 "writting" should be updated to "writing";
 "destory" should be updated to "destroy";
 "ture" should be updated to "true";
 "interupt" should be updated to "interrupt";




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Permissions to edit Confluence Wiki

Re: Permissions to edit Confluence Wiki

Re: [DISCUSS] Merge yarn-native-services branch into trunk

Re: [VOTE] Merge yarn-native-services branch into trunk

Permissions to edit Confluence Wiki

[jira] [Resolved] (HADOOP-14650) Update jsp-api version

[jira] [Resolved] (HADOOP-14656) Update xercesImpl version to 2.11.0

[jira] [Created] (HADOOP-14855) Hadoop scripts may errantly believe a daemon is still running, preventing it from starting

[jira] [Created] (HADOOP-14854) DistCp should not issue file status calls for files in the filter list

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[jira] [Created] (HADOOP-14853) hadoop-mapreduce-client-app is not a client module

Re: [VOTE] Merge yarn-native-services branch into trunk

[jira] [Created] (HADOOP-14852) Intermittent failure of S3Guard TestConsistencyListFiles

[jira] [Reopened] (HADOOP-9383) mvn clean compile fails without install goal

[jira] [Resolved] (HADOOP-8391) Hadoop-auth should use log4j

[DISCUSS] official docker image(s) for hadoop

[jira] [Resolved] (HADOOP-9383) mvn clean compile fails without install goal

hadoop roadmaps

[jira] [Created] (HADOOP-14851) LambdaTestUtils.eventually() doesn't spin on Assertion failures

[jira] [Created] (HADOOP-14850) Read HttpServer2 resources directly from the source tree (if exists)

Re: 2017-09-07 Hadoop 3 release status update

[jira] [Created] (HADOOP-14849) some wrong spelling words update

22 matches

Site Navigation

Mail list logo

Footer information