[jira] [Created] (HADOOP-14188) Remove the usage of org.mockito.internal.util.reflection.Whitebox

2017-03-15 Thread Akira Ajisaka (JIRA) 
Akira Ajisaka created HADOOP-14188:
--

 Summary: Remove the usage of 
org.mockito.internal.util.reflection.Whitebox
 Key: HADOOP-14188
 URL: https://issues.apache.org/jira/browse/HADOOP-14188
 Project: Hadoop Common
  Issue Type: Test
  Components: test
Reporter: Akira Ajisaka
Priority: Minor


org.mockito.internal.util.reflection.Whitebox was removed in Mockito 2.1, so we 
need to remove the usage to upgrade Mockito. Getter/setter method can be used 
instead of this hack.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14187) Update ZooKeeper dependency to 3.4.8 and Curator dependency to 2.12.0

2017-03-15 Thread Tsuyoshi Ozawa (JIRA) 
Tsuyoshi Ozawa created HADOOP-14187:
---

 Summary: Update ZooKeeper dependency to 3.4.8 and Curator 
dependency to 2.12.0
 Key: HADOOP-14187
 URL: https://issues.apache.org/jira/browse/HADOOP-14187
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Tsuyoshi Ozawa


This is a update for using Apache Curator which shades guava.

Why is Curator updated to 2.12.0 instead of 3.3.0?
It's because Curator 3.x only supports ZooKeeper 3.5.x. ZooKeeper 3.5.x is 
still alpha release. Hence, I think we should do conservative choice here.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

2017-03-15 Thread Andrew Wang 
On Wed, Mar 15, 2017 at 5:42 PM, Junping Du  wrote:

> Hi Eric,
>  Thanks for your verification work! About your question on RM's key,
> we actually mentioned we were using  https://dist.apache.org/repos/
> dist/release/hadoop/common/KEYS in our hadoop wiki page:
> https://wiki.apache.org/hadoop/HowToRelease. Also, for hadoop user, our
> release page (http://hadoop.apache.org/releases.html) points key file
> location to the same place. So for developers and users in hadoop
> community, I hope this is not confusing too much.
>  However, from my offline check with Owen, it sounds like
> http://home.apache.org/keys/group/hadoop.asc is something tradition for
> apache projects and convenient for usage. I already updated related key to
> my apache id which should sync to there automatically. We'd better document
> it also in our hadoop wiki page.
>
> I actually asked INFRA about this when I was adding my key, a little more
backstory:

We used to have a README in dist saying to add your key on id.apache.org,
then to export the hadoop group's keys to generate dist's KEYS file.

INFRA told me this is a Bad Thing, since the KEYS file should be append
only. This way, users can still verify a release even if an RM leaves the
hadoop group or changes their key on id.apache.org.

So, I deleted the old README instructions. The dist KEYS file is the
canonical (and only) place to look for an RM's keys. Based on Junping's
examination, it sounds like our docs to reflect this. I'd rather not
complicate matters by also discussing the hadoop group's keys.

Best,
Andrew

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

2017-03-15 Thread Junping Du 
Hi Eric,
 Thanks for your verification work! About your question on RM's key, we 
actually mentioned we were using  
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS in our hadoop 
wiki page: https://wiki.apache.org/hadoop/HowToRelease. Also, for hadoop user, 
our release page (http://hadoop.apache.org/releases.html) points key file 
location to the same place. So for developers and users in hadoop community, I 
hope this is not confusing too much.
 However, from my offline check with Owen, it sounds like 
http://home.apache.org/keys/group/hadoop.asc is something tradition for apache 
projects and convenient for usage. I already updated related key to my apache 
id which should sync to there automatically. We'd better document it also in 
our hadoop wiki page.  

Thanks,

Junping

From: Eric Badger 
Sent: Wednesday, March 15, 2017 2:06 PM
To: Junping Du; Steve Loughran
Cc: common-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

All on MacOS Sierra

Verified signatures
  - Minor note: Junping, I had a hard time finding your key. I grabbed the keys 
for hadoop from
http://home.apache.org/keys/group/hadoop.asc and you had a key there, but it 
wasn't the one that you signed this commit with. Then with some help from Jason 
I found the correct key at
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS. So it would be 
nice if those were in sync.
Compiled from source
Deployed pseudo-distributed cluster
Ran some sample MR jobs

+1 (non-binding)

Thanks,

Eric


On Wednesday, March 15, 2017 2:58 PM, Junping Du  wrote:



The latest commit on RC2 is: e51312e8e106efb2ebd4844eecacb51026fac8b7.
btw, I think tags are immutable. Isn't it?

Thanks,

Junping


From: Steve Loughran
Sent: Wednesday, March 15, 2017 12:30 PM
To: Junping Du
Cc: common-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

> On 14 Mar 2017, at 08:41, Junping Du  wrote:
>
> Hi all,
> With several important fixes get merged last week, I've created a new 
> release candidate (RC2) for Apache Hadoop 2.8.0.
>
> This is the next minor release to follow up 2.7.0 which has been released 
> for more than 1 year. It comprises 2,919 fixes, improvements, and new 
> features. Most of these commits are released for the first time in branch-2.
>
>  More information about the 2.8.0 release plan can be found here: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>
>  Please note that RC0 and RC1 are not voted public because significant 
> issues are found just after RC tag getting published.
>
>  The RC is available at: 
> http://home.apache.org/~junping_du/hadoop-2.8.0-RC2
>
>  The RC tag in git is: release-2.8.0-RC2

given tags are so easy to move, we need to be relying on one or more of:
-the commit ID,
-the tag being signed

Junping: what is the commit Id for the release?

>
>  The maven artifacts are available via repository.apache.org at: 
> https://repository.apache.org/content/repositories/orgapachehadoop-1056
>

thanks, I'll play with these downstream, as well as checking out and trying to 
build on windows

>  Please try the release and vote; the vote will run for the usual 5 days, 
> ending on 03/20/2017 PDT time.
>
> Thanks,
>
> Junping

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

2017-03-15 Thread Junping Du 
bq. From my read of the poms, hadoop-client depends on hadoop-hdfs-client to 
pull in HDFS-related code. It doesn't have its own dependency on hadoop-hdfs. 
So I think this affects users of the hadoop-client artifact, which has existed 
for a long time.

I could miss that. Thanks for reminding! From my quick check: 
https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client/2.7.3?, it 
sounds like 669 artifacts from other projects were depending on it.


I think we should withdraw the current RC bits. Please stop the verification & 
vote.

I will kick off another RC immediately when HDFS-11431 get fixed.


Thanks,


Junping



From: Andrew Wang 
Sent: Wednesday, March 15, 2017 2:04 PM
To: Junping Du
Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

Hi Junping, inline,


>From my understanding, this issue is related to our previous improvements with 
>separating client and server jars in HDFS-6200. If we use the new "client" jar 
>in NN HA deployment, then we will hit the issue reported.

>From my read of the poms, hadoop-client depends on hadoop-hdfs-client to pull 
>in HDFS-related code. It doesn't have its own dependency on hadoop-hdfs. So I 
>think this affects users of the hadoop-client artifact, which has existed for 
>a long time.

Essentially all of our customer deployments run with NN HA, so this would 
affect a lot of users.

I can see two options here:

- Without any change in 2.8.0, if user hit the issue when they deploy HA 
cluster by using new client jar, adding back hdfs jar just like how things work 
previously

- Make the change now in 2.8.0, either moving ConfiguredFailoverProxyProvider 
to client jar or adding dependency between client jar and server jar. There 
must be some arguments there on which way to fix is better especially 
ConfiguredFailoverProxyProvider still has some sever side dependencies.


I would prefer the first option, given:

- The issue fixing time is unpredictable as there are still discussion on how 
to fix this issue. Our 2.8.0 release shouldn't be an endless journey which has 
been deferred several times for more serious issue.

Looks like we have a patch being actively revved and reviewed to fix this by 
making hadoop-hdfs-client depend on hadoop-hdfs. Thanks to Steven and Steve for 
working on this.

Steve proposed doing a proper split in a later JIRA.

- We have workaround for this improvement, no regression happens due to this 
issue. People can still use hdfs jar in old way. The worst case is improvement 
for HDFS doesn't work in some cases - that shouldn't block the whole release.

Based on the above, I think there is a regression for users of the 
hadoop-client artifact.

If it actually only affects users of hadoop-hdfs-client, then I agree we can 
document it as a Known Issue and fix it later.

Best,
Andrew

[jira] [Created] (HADOOP-14186) Make messages in AzureFileSystemThreadPoolExecutor debug

2017-03-15 Thread Subramanyam Pattipaka (JIRA) 
Subramanyam Pattipaka created HADOOP-14186:
--

 Summary: Make messages in AzureFileSystemThreadPoolExecutor debug
 Key: HADOOP-14186
 URL: https://issues.apache.org/jira/browse/HADOOP-14186
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Subramanyam Pattipaka


Some of warn messages are confusing to end users if they disable parallism. 
Move them to debug.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

2017-03-15 Thread Josh Elser 
A tag is immutable, but you (or someone else) could remove the tag you 
pushed and re-push a new one. That's why the commit id is important -- 
it ensures that everyone else knows the exact commit being voted on.


Junping Du wrote:

The latest commit on RC2 is: e51312e8e106efb2ebd4844eecacb51026fac8b7.
btw, I think tags are immutable. Isn't it?

Thanks,

Junping

From: Steve Loughran
Sent: Wednesday, March 15, 2017 12:30 PM
To: Junping Du
Cc: common-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)


On 14 Mar 2017, at 08:41, Junping Du  wrote:

Hi all,
 With several important fixes get merged last week, I've created a new 
release candidate (RC2) for Apache Hadoop 2.8.0.

 This is the next minor release to follow up 2.7.0 which has been released 
for more than 1 year. It comprises 2,919 fixes, improvements, and new features. 
Most of these commits are released for the first time in branch-2.

  More information about the 2.8.0 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  Please note that RC0 and RC1 are not voted public because significant 
issues are found just after RC tag getting published.

  The RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.0-RC2

  The RC tag in git is: release-2.8.0-RC2


given tags are so easy to move, we need to be relying on one or more of:
-the commit ID,
-the tag being signed

Junping: what is the commit Id for the release?


  The maven artifacts are available via repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1056



thanks, I'll play with these downstream, as well as checking out and trying to 
build on windows


  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 03/20/2017 PDT time.

Thanks,

Junping


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-13959) S3guard: replace dynamo.describe() call in init with more efficient query

2017-03-15 Thread Mingliang Liu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu resolved HADOOP-13959.

Resolution: Fixed

As [HADOOP-13985] is committed, I think we can resolve this one. Feel free to 
re-open if necessary.

> S3guard: replace dynamo.describe() call in init with more efficient query
> -
>
> Key: HADOOP-13959
> URL: https://issues.apache.org/jira/browse/HADOOP-13959
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Steve Loughran
>Assignee: Mingliang Liu
>Priority: Minor
>
> HADOOP-13908 adds initialization when a table isn't created, using the 
> {{describe()}} call.
> AWS document this as inefficient, and throttle it. We should be able to get 
> away with a simple table lookup as the probe



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

2017-03-15 Thread Eric Badger 
All on MacOS Sierra

Verified signatures
  - Minor note: Junping, I had a hard time finding your key. I grabbed the keys 
for hadoop from 
http://home.apache.org/keys/group/hadoop.asc and you had a key there, but it 
wasn't the one that you signed this commit with. Then with some help from Jason 
I found the correct key at 
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS. So it would be 
nice if those were in sync. 
Compiled from source
Deployed pseudo-distributed cluster
Ran some sample MR jobs

+1 (non-binding)

Thanks,

Eric


On Wednesday, March 15, 2017 2:58 PM, Junping Du  wrote:



The latest commit on RC2 is: e51312e8e106efb2ebd4844eecacb51026fac8b7.
btw, I think tags are immutable. Isn't it?

Thanks,

Junping


From: Steve Loughran
Sent: Wednesday, March 15, 2017 12:30 PM
To: Junping Du
Cc: common-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

> On 14 Mar 2017, at 08:41, Junping Du  wrote:
>
> Hi all,
> With several important fixes get merged last week, I've created a new 
> release candidate (RC2) for Apache Hadoop 2.8.0.
>
> This is the next minor release to follow up 2.7.0 which has been released 
> for more than 1 year. It comprises 2,919 fixes, improvements, and new 
> features. Most of these commits are released for the first time in branch-2.
>
>  More information about the 2.8.0 release plan can be found here: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>
>  Please note that RC0 and RC1 are not voted public because significant 
> issues are found just after RC tag getting published.
>
>  The RC is available at: 
> http://home.apache.org/~junping_du/hadoop-2.8.0-RC2
>
>  The RC tag in git is: release-2.8.0-RC2

given tags are so easy to move, we need to be relying on one or more of:
-the commit ID,
-the tag being signed

Junping: what is the commit Id for the release?

>
>  The maven artifacts are available via repository.apache.org at: 
> https://repository.apache.org/content/repositories/orgapachehadoop-1056
>

thanks, I'll play with these downstream, as well as checking out and trying to 
build on windows

>  Please try the release and vote; the vote will run for the usual 5 days, 
> ending on 03/20/2017 PDT time.
>
> Thanks,
>
> Junping

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

2017-03-15 Thread Andrew Wang 
Hi Junping, inline,

>From my understanding, this issue is related to our previous
> improvements with separating client and server jars in HDFS-6200. If we use
> the new "client" jar in NN HA deployment, then we will hit the issue
> reported.
>
>From my read of the poms, hadoop-client depends on hadoop-hdfs-client to
pull in HDFS-related code. It doesn't have its own dependency on
hadoop-hdfs. So I think this affects users of the hadoop-client artifact,
which has existed for a long time.

Essentially all of our customer deployments run with NN HA, so this would
affect a lot of users.

> I can see two options here:
>
> - Without any change in 2.8.0, if user hit the issue when they deploy HA
> cluster by using new client jar, adding back hdfs jar just like how things
> work previously
>
> - Make the change now in 2.8.0, either moving
> ConfiguredFailoverProxyProvider to client jar or adding dependency
> between client jar and server jar. There must be some arguments there on
> which way to fix is better especially ConfiguredFailoverProxyProvider
> still has some sever side dependencies.
>
>
> I would prefer the first option, given:
>
> - The issue fixing time is unpredictable as there are still discussion on
> how to fix this issue. Our 2.8.0 release shouldn't be an endless journey
> which has been deferred several times for more serious issue.
>
Looks like we have a patch being actively revved and reviewed to fix this
by making hadoop-hdfs-client depend on hadoop-hdfs. Thanks to Steven and
Steve for working on this.

Steve proposed doing a proper split in a later JIRA.

> - We have workaround for this improvement, no regression happens due to
> this issue. People can still use hdfs jar in old way. The worst case
> is improvement for HDFS doesn't work in some cases - that shouldn't block
> the whole release.
>
Based on the above, I think there is a regression for users of the
hadoop-client artifact.

If it actually only affects users of hadoop-hdfs-client, then I agree we
can document it as a Known Issue and fix it later.

Best,
Andrew

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

2017-03-15 Thread Junping Du 
The latest commit on RC2 is: e51312e8e106efb2ebd4844eecacb51026fac8b7.
btw, I think tags are immutable. Isn't it?

Thanks,

Junping

From: Steve Loughran
Sent: Wednesday, March 15, 2017 12:30 PM
To: Junping Du
Cc: common-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

> On 14 Mar 2017, at 08:41, Junping Du  wrote:
>
> Hi all,
> With several important fixes get merged last week, I've created a new 
> release candidate (RC2) for Apache Hadoop 2.8.0.
>
> This is the next minor release to follow up 2.7.0 which has been released 
> for more than 1 year. It comprises 2,919 fixes, improvements, and new 
> features. Most of these commits are released for the first time in branch-2.
>
>  More information about the 2.8.0 release plan can be found here: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>
>  Please note that RC0 and RC1 are not voted public because significant 
> issues are found just after RC tag getting published.
>
>  The RC is available at: 
> http://home.apache.org/~junping_du/hadoop-2.8.0-RC2
>
>  The RC tag in git is: release-2.8.0-RC2

given tags are so easy to move, we need to be relying on one or more of:
-the commit ID,
-the tag being signed

Junping: what is the commit Id for the release?

>
>  The maven artifacts are available via repository.apache.org at: 
> https://repository.apache.org/content/repositories/orgapachehadoop-1056
>

thanks, I'll play with these downstream, as well as checking out and trying to 
build on windows

>  Please try the release and vote; the vote will run for the usual 5 days, 
> ending on 03/20/2017 PDT time.
>
> Thanks,
>
> Junping

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le

2017-03-15 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/258/

[Mar 14, 2017 6:47:25 PM] (wang) HDFS-11505. Do not enable any erasure coding 
policies by default.
[Mar 14, 2017 7:52:25 PM] (naganarasimha_gr) YARN-6327. Removing queues from 
CapacitySchedulerQueueManager and
[Mar 14, 2017 7:58:12 PM] (junping_du) YARN-6313. YARN logs cli should provide 
logs for a completed container
[Mar 14, 2017 8:03:42 PM] (liuml07) Revert "HADOOP-14170. 
FileSystemContractBaseTest is not cleaning up test
[Mar 14, 2017 9:38:21 PM] (liuml07) HADOOP-14170. FileSystemContractBaseTest is 
not cleaning up test
[Mar 14, 2017 10:09:47 PM] (rchiang) YARN-6331. Fix flakiness in 
TestFairScheduler#testDumpState. (Yufei Gu
[Mar 14, 2017 11:41:10 PM] (wang) HDFS-9705. Refine the behaviour of 
getFileChecksum when length = 0.
[Mar 15, 2017 9:18:05 AM] (sunilg) YARN-6328. Fix a spelling mistake in 
CapacityScheduler. Contributed by
[Mar 15, 2017 10:05:03 AM] (yqlin) HDFS-11420. Edit file should not be 
processed by the same type processor
[Mar 15, 2017 10:24:09 AM] (rohithsharmaks) YARN-6336. Jenkins report YARN new 
UI build failure. Contributed by




-1 overall


The following subsystems voted -1:
compile unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.TestFileChecksum 
   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.server.timeline.TestRollingLevelDB 
   hadoop.yarn.server.timeline.TestTimelineDataManager 
   hadoop.yarn.server.timeline.TestLeveldbTimelineStore 
   hadoop.yarn.server.timeline.webapp.TestTimelineWebServices 
   hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore 
   hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore 
   
hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer 
   hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore 
   
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption 
   hadoop.yarn.server.resourcemanager.TestRMRestart 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore 
   hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient 
   hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.mapred.TestShuffleHandler 
   hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService 

Timed out junit tests :

   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean 
   org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache 
  

   compile:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/258/artifact/out/patch-compile-root.txt
  [136K]

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/258/artifact/out/patch-compile-root.txt
  [136K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/258/artifact/out/patch-compile-root.txt
  [136K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/258/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [276K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/258/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/258/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [52K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/258/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [72K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/258/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [324K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/258/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt
  [28K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/258/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
  [12K]

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-03-15 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/346/

[Mar 14, 2017 9:49:48 AM] (yqlin) HDFS-11526. Fix confusing block recovery 
message. Contributed by Yiqun
[Mar 14, 2017 9:58:07 AM] (junping_du) YARN-6314. Potential infinite 
redirection on YARN log redirection web
[Mar 14, 2017 6:47:25 PM] (wang) HDFS-11505. Do not enable any erasure coding 
policies by default.
[Mar 14, 2017 7:52:25 PM] (naganarasimha_gr) YARN-6327. Removing queues from 
CapacitySchedulerQueueManager and
[Mar 14, 2017 7:58:12 PM] (junping_du) YARN-6313. YARN logs cli should provide 
logs for a completed container
[Mar 14, 2017 8:03:42 PM] (liuml07) Revert "HADOOP-14170. 
FileSystemContractBaseTest is not cleaning up test
[Mar 14, 2017 9:38:21 PM] (liuml07) HADOOP-14170. FileSystemContractBaseTest is 
not cleaning up test
[Mar 14, 2017 10:09:47 PM] (rchiang) YARN-6331. Fix flakiness in 
TestFairScheduler#testDumpState. (Yufei Gu
[Mar 14, 2017 11:41:10 PM] (wang) HDFS-9705. Refine the behaviour of 
getFileChecksum when length = 0.




-1 overall


The following subsystems voted -1:
asflicense compile unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.server.datanode.TestDataNodeLifeline 
   hadoop.yarn.server.nodemanager.containermanager.TestContainerManager 
   hadoop.yarn.server.timeline.webapp.TestTimelineWebServices 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.mapred.TestMRTimelineEventHandling 
  

   compile:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/346/artifact/out/patch-compile-root.txt
  [208K]

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/346/artifact/out/patch-compile-root.txt
  [208K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/346/artifact/out/patch-compile-root.txt
  [208K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/346/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/346/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/346/artifact/out/diff-patch-shellcheck.txt
  [24K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/346/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/346/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/346/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/346/artifact/out/diff-javadoc-javadoc-root.txt
  [2.2M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/346/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [264K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/346/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [36K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/346/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/346/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [56K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/346/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [324K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/346/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-ui.txt
  [32K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/346/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
  [88K]

   asflicense:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/346/artifact/out/patch-asflicense-problems.txt
  [4.0K]

Powered by Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

2017-03-15 Thread Steve Loughran 

> On 15 Mar 2017, at 00:36, Junping Du  wrote:
> 
> Thanks Andrew for reporting the issue. This JIRA is out of my radar as it? 
> didn't specify any target version before.
> 
> 
> From my understanding, this issue is related to our previous improvements 
> with separating client and server jars in HDFS-6200. If we use the new 
> "client" jar in NN HA deployment, then we will hit the issue reported.
> 
> 
> I can see two options here:
> 
> - Without any change in 2.8.0, if user hit the issue when they deploy HA 
> cluster by using new client jar, adding back hdfs jar just like how things 
> work previously
> 
> - Make the change now in 2.8.0, either moving ConfiguredFailoverProxyProvider 
> to client jar or adding dependency between client jar and server jar. There 
> must be some arguments there on which way to fix is better especially 
> ConfiguredFailoverProxyProvider still has some sever side dependencies.
> 
> 
> I would prefer the first option, given:
> 
> - The issue fixing time is unpredictable as there are still discussion on how 
> to fix this issue. Our 2.8.0 release shouldn't be an endless journey which 
> has been deferred several times for more serious issue.
> 
> - We have workaround for this improvement, no regression happens due to this 
> issue. People can still use hdfs jar in old way. The worst case is 
> improvement for HDFS doesn't work in some cases - that shouldn't block the 
> whole release.
> 
> 
> I think we should let vote keep going unless someone have more concerns which 
> I could miss.

getting it out the door with this in the release notes, and a plan for 2.8.1 
would be ideal

> 
> 
> 
> Thanks,
> 
> 
> Junping
> 
> 
> 
> From: Andrew Wang 
> Sent: Tuesday, March 14, 2017 2:50 PM
> To: Junping Du
> Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
> yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)
> 
> Hi Junping,
> 
> Noticed this possible blocker float by my inbox today. It had an affects but 
> no target version set:
> 
> https://issues.apache.org/jira/browse/HDFS-11431
> 
> Thoughts? Seems like the hadoop-hdfs-client artifact doesn't work right now.
> 
> Best,
> Andrew
> 
> 
> On Tue, Mar 14, 2017 at 1:41 AM, Junping Du 
> > wrote:
> Hi all,
> With several important fixes get merged last week, I've created a new 
> release candidate (RC2) for Apache Hadoop 2.8.0.
> 
> This is the next minor release to follow up 2.7.0 which has been released 
> for more than 1 year. It comprises 2,919 fixes, improvements, and new 
> features. Most of these commits are released for the first time in branch-2.
> 
>  More information about the 2.8.0 release plan can be found here: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
> 
>  Please note that RC0 and RC1 are not voted public because significant 
> issues are found just after RC tag getting published.
> 
>  The RC is available at: 
> http://home.apache.org/~junping_du/hadoop-2.8.0-RC2
> 
>  The RC tag in git is: release-2.8.0-RC2
> 
>  The maven artifacts are available via 
> repository.apache.org at: 
> https://repository.apache.org/content/repositories/orgapachehadoop-1056
> 
>  Please try the release and vote; the vote will run for the usual 5 days, 
> ending on 03/20/2017 PDT time.
> 
> Thanks,
> 
> Junping
> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14185) Remove service loader config entry for Har fs

2017-03-15 Thread Steve Loughran (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-14185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HADOOP-14185.
-
Resolution: Won't Fix

closing as a wontfix then.

john, you could have just done a "cut back on service declarations" JIRA and 
done them all in one go...

> Remove service loader config entry for Har fs
> -
>
> Key: HADOOP-14185
> URL: https://issues.apache.org/jira/browse/HADOOP-14185
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.7.3
>Reporter: John Zhuge
>Priority: Minor
>  Labels: newbie
>
> Per discussion in HADOOP-14132. Remove line 
> {{org.apache.hadoop.fs.HarFileSystem}} from the service loader config file 
> hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
>  and add property {{fs.har.impl}} to {{core-default.xml}}. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14185) Remove service loader config file for Har fs

2017-03-15 Thread John Zhuge (JIRA) 
John Zhuge created HADOOP-14185:
---

 Summary: Remove service loader config file for Har fs
 Key: HADOOP-14185
 URL: https://issues.apache.org/jira/browse/HADOOP-14185
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Affects Versions: 2.7.3
Reporter: John Zhuge
Priority: Minor


Per discussion in HADOOP-14132. Remove line 
{{org.apache.hadoop.fs.HarFileSystem}} from the service loader config file 
hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
 and add property {{fs.har.impl}} to {{core-default.xml}}. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14184) Remove service loader config file for ftp fs

2017-03-15 Thread John Zhuge (JIRA) 
John Zhuge created HADOOP-14184:
---

 Summary: Remove service loader config file for ftp fs
 Key: HADOOP-14184
 URL: https://issues.apache.org/jira/browse/HADOOP-14184
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Affects Versions: 2.7.3
Reporter: John Zhuge
Priority: Minor


Per discussion in HADOOP-14132. Remove line 
{{org.apache.hadoop.fs.ftp.FTPFileSystem}} from the service loader config file 
hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
 and add property {{fs.ftp.impl}} to {{core-default.xml}}. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14183) No service loader for wasb fs

2017-03-15 Thread John Zhuge (JIRA) 
John Zhuge created HADOOP-14183:
---

 Summary: No service loader for wasb fs
 Key: HADOOP-14183
 URL: https://issues.apache.org/jira/browse/HADOOP-14183
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs/azure
Affects Versions: 2.7.3
Reporter: John Zhuge
Priority: Minor


Per discussion in HADOOP-14132. Remove the service loader config file 
hadoop-tools/hadoop-azure/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
 and add property {{fs.wasb.impl}} to {{core-default.xml}}. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org