[jira] [Created] (HADOOP-13376) How to cross compile hadoop on buildroot

[Yadong Zhang (JIRA)]

Yadong Zhang created HADOOP-13376:
-

 Summary: How to cross compile hadoop on buildroot
 Key: HADOOP-13376
 URL: https://issues.apache.org/jira/browse/HADOOP-13376
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Affects Versions: 2.2.0
 Environment: Ubuntu 14.04 x86_64
Hadoop 2.2.0
Buildroot 2015.05


Reporter: Yadong Zhang


I need to run hadoop on IBM powerpc64le arch server, and I modified the Hadoop 
source code, But I want to compile it on my own dev env (Ubuntu 14.04 x86_64). 
As I know, they are different arch, the package compiled on my own machine can 
not run on IBM server. So I want to use cross compile tools Buildroot. But I do 
not know how to use it to compile hadoop. Can any one to help me?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli]

HADOOP-12893 is done after 4 months, thanks to great work from a bunch of folks 
besides Xiao Chen.

Creating the RC now.

Thanks
+Vinod

> On Jun 14, 2016, at 7:28 PM, Vinod Kumar Vavilapalli  
> wrote:
> 
> Release branch 2.7.3 is created. I also updated branch-2.7 to point to 
> 2.7.4-SNAPSHOT now.
> 
> Thanks
> +Vinod
> 
>> On Jun 14, 2016, at 3:54 PM, Vinod Kumar Vavilapalli  
>> wrote:
>> 
>> HADOOP-12893 is finally close to completion after >  3months thanks to 
>> efforts from Akira AJISAKA, Xiao Chen and Andrew Wang.
>> 
>> I’m creating the release branch and kickstarting the release activities.
>> 
>> Thanks
>> +Vinod
>> 
>>> On May 16, 2016, at 5:39 PM, Vinod Kumar Vavilapalli  
>>> wrote:
>>> 
>>> I am just waiting on HADOOP-12893.
>>> 
>>> HADOOP-13154 just got created in the last one day, will have to see if it 
>>> really should block the release.
>>> 
>>> Major tickets are usually taken on a time basis: if they get in by the 
>>> proposed timelines, we get them in. Otherwise, we move them over.
>>> 
>>> Thanks
>>> +Vinod
>>> 
 On May 16, 2016, at 5:20 PM, larry mccay  wrote:
 
 Curious on the status of 2.7.3
 
 It seems that we still have two outstanding critical/blocker JIRAs:
 
 1. [image: Bug] HADOOP-12893
 Verify LICENSE.txt and NOTICE.txt
 
 2. [image: Sub-task] HADOOP-13154
 S3AFileSystem
 printAmazonServiceException/printAmazonClientException appear copy & paste
 of AWS examples 
 
 
 But 45-ish when we include Majors as well.
 
 I know there are a number of critical issues with fixes that need to go 
 out.
 
 What is the plan?
 
 On Tue, Apr 12, 2016 at 2:09 PM, Vinod Kumar Vavilapalli 
  wrote:
 
> Others and I committed a few, I pushed out a few.
> 
> Down to just three now!
> 
> +Vinod
> 
>> On Apr 6, 2016, at 3:00 PM, Vinod Kumar Vavilapalli 
> wrote:
>> 
>> Down to only 10 blocker / critical tickets (
> https://issues.apache.org/jira/issues/?filter=12335343 <
> https://issues.apache.org/jira/issues/?filter=12335343>) now!
>> 
>> Thanks
>> +Vinod
>> 
>>> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org > wrote:
>>> 
>>> Hi all,
>>> 
>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
> (which did go out mid February). Got a little busy since.
>>> 
>>> Following up the 2.7.2 maintenance release, we should work towards a
> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and
> *no* features / improvements.
>>> 
>>> I hope to cut an RC in a week - giving enough time for outstanding
> blocker / critical issues. Will start moving out any tickets that are not
> blockers and/or won’t fit the timeline - there are 3 blockers and 15
> critical tickets outstanding as of now.
>>> 
>>> Thanks,
>>> +Vinod
>>> 
>>> [1] 2.7.3 release blockers:
> https://issues.apache.org/jira/issues/?filter=12335343 <
> https://issues.apache.org/jira/issues/?filter=12335343>
>> 
> 
> 
>>> 
>>> 
>>> -
>>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>> 
>>> 
>> 
> 
> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Feedback on IRC channel

[Sangjin Lee]

I seldom check out IRC (as my experience was the same). I'm OK with
retiring it if no committers are around.

On a related note, I know Tsuyoshi set up a slack channel for the
committers. Even that one is pretty idle. :) Should we use it more often?
If that starts to gain traction, we could set up a more open room for users
as well.

Sangjin

On Wed, Jul 13, 2016 at 9:13 AM, Karthik Kambatla 
wrote:

> Recently, Andrew Wang and I were at an academic conference where one of the
> attendees (a grad student) was mentioning that his posts to the IRC channel
> are never answered.
>
> Personally, I haven't been using the IRC channel. Neither do I know anyone
> who is actively monitoring it.
>
> I am emailing to check:
>
>1. Are there folks actively monitoring the IRC channel and answering
>questions?
>2. If there is no one, should we consider retiring the channel?
>
> Thanks
> Karthik
>

[jira] [Created] (HADOOP-13374) Add the L verification script

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13374:
--

 Summary: Add the L verification script
 Key: HADOOP-13374
 URL: https://issues.apache.org/jira/browse/HADOOP-13374
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Xiao Chen
Assignee: Xiao Chen


This is the script that's used for L change verification during HADOOP-12893. 
We should commit this as [~ozawa] 
[suggested|https://issues.apache.org/jira/browse/HADOOP-13298?focusedCommentId=15374498=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15374498].

I was 
[initially|https://issues.apache.org/jira/browse/HADOOP-12893?focusedCommentId=15283040=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15283040]
 verifying from an on-fly shell command, and [~andrew.wang] contributed the 
script later in [a comment|
https://issues.apache.org/jira/browse/HADOOP-12893?focusedCommentId=15303281=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15303281],
 so most credit should go to him. :)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13373) Add S3A implementation of FSMainOperationsBaseTest

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-13373:
---

 Summary: Add S3A implementation of FSMainOperationsBaseTest
 Key: HADOOP-13373
 URL: https://issues.apache.org/jira/browse/HADOOP-13373
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3
Affects Versions: 2.8.0
Reporter: Steve Loughran
Priority: Minor


There's a JUnit 4 test suite, {{FSMainOperationsBaseTest}}, which should be 
implemented in the s3a tests, to add a bit more test coverage —including for 
globbing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13372) MR jobs can not access Swift filesystem if Kerberos is enabled

[ramtin (JIRA)]

ramtin created HADOOP-13372:
---

 Summary: MR jobs can not access Swift filesystem if Kerberos is 
enabled
 Key: HADOOP-13372
 URL: https://issues.apache.org/jira/browse/HADOOP-13372
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs, fs/swift, security
Reporter: ramtin






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Feedback on IRC channel

[Karthik Kambatla]

Recently, Andrew Wang and I were at an academic conference where one of the
attendees (a grad student) was mentioning that his posts to the IRC channel
are never answered.

Personally, I haven't been using the IRC channel. Neither do I know anyone
who is actively monitoring it.

I am emailing to check:

   1. Are there folks actively monitoring the IRC channel and answering
   questions?
   2. If there is no one, should we consider retiring the channel?

Thanks
Karthik

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/93/

[Jul 12, 2016 3:11:52 PM] (varunsaxena) YARN-5354. 
TestDistributedShell.checkTimelineV2 fails randomly for
[Jul 12, 2016 3:15:53 PM] (varunsaxena) MAPREDUCE-6731.
[Jul 12, 2016 3:29:49 PM] (junping_du) New addendum to YARN-5080. Cannot obtain 
logs using YARN CLI -am for
[Jul 12, 2016 8:27:41 PM] (jlowe) YARN-5317. 
testAMRestartNotLostContainerCompleteMsg may fail.
[Jul 12, 2016 10:12:25 PM] (rkanter) YARN-4366. Fix Lint Warnings in YARN 
Common (templedf via rkanter)
[Jul 12, 2016 10:51:18 PM] (rkanter) MAPREDUCE-6625. TestCLI#testGetJob fails 
occasionally (haibochen via
[Jul 12, 2016 11:16:03 PM] (lei) HDFS-10387. DataTransferProtocol#writeBlock 
missing some javadocs (John
[Jul 12, 2016 11:49:32 PM] (lei) HADOOP-13315. FileContext#umask is not 
initialized properly. (John Zhuge
[Jul 13, 2016 2:25:34 AM] (varunsaxena) YARN-5353. ResourceManager can leak 
delegation tokens when they are
[Jul 13, 2016 6:18:37 AM] (zhz) HDFS-10544. Balancer doesn't work with 
IPFailoverProxyProvider.




-1 overall


The following subsystems voted -1:
unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation 
   
hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
 
   hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.client.cli.TestLogsCLI 
   hadoop.yarn.client.api.impl.TestYarnClient 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/93/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/93/artifact/out/diff-compile-javac-root.txt
  [172K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/93/artifact/out/diff-checkstyle-root.txt
  [16M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/93/artifact/out/diff-patch-pylint.txt
  [16K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/93/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/93/artifact/out/diff-patch-shelldocs.txt
  [16K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/93/artifact/out/whitespace-eol.txt
  [12M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/93/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/93/artifact/out/diff-javadoc-javadoc-root.txt
  [2.3M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/93/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [240K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/93/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [40K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/93/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [56K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/93/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [268K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/93/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/93/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt
  [124K]

Powered by Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13371) S3A globber to use bulk listObject call over recursive directory scan

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-13371:
---

 Summary: S3A globber to use bulk listObject call over recursive 
directory scan
 Key: HADOOP-13371
 URL: https://issues.apache.org/jira/browse/HADOOP-13371
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs, fs/s3
Affects Versions: 2.8.0
Reporter: Steve Loughran


HADOOP-13208 produces O(1) listing of directory trees in 
{{FileSystem.listStatus}} calls, but doesn't do anything for 
{{FileSystem.globStatus()}}, which uses a completely different codepath, one 
which does a selective recursive scan by pattern matching as it goes down, 
filtering out those patterns which don't match. Cost is O(matching-directories) 
+ cost of examining the files.

It should be possible to do the glob status listing in S3A not through the 
filtered treewalk, but through a list + filter operation. This would be an 
O(files) lookup *before any filtering took place*.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-13370) After creating HA, getting java.net.UnknownHostException

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HADOOP-13370.
-
Resolution: Invalid

JIRA isn't a support channel, sorry

https://wiki.apache.org/hadoop/InvalidJiraIssues

> After creating HA, getting java.net.UnknownHostException
> 
>
> Key: HADOOP-13370
> URL: https://issues.apache.org/jira/browse/HADOOP-13370
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 2.6.4
>Reporter: Bhavan Jindal
>Priority: Critical
>
> I have made my standalone cluster to a high available using quorum journal 
> method. After that i am not able to access HDFS. Getting below error :
> [hadoop@namenode hadoop]$ hdfs dfs -ls /
> 16/07/12 22:46:05 WARN util.NativeCodeLoader: Unable to load native-hadoop 
> library for your platform... using builtin-java classes where applicable
> -ls: java.net.UnknownHostException: myhahdpcluster
> Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [ ...]
> [hadoop@namenode hadoop]$
> All the other HA components are working fine. Can you please let me know if i 
> am missing anything.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org