date:20180410

Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64

2018-04-10 Thread Apache Jenkins Server

For more details, see https://builds.apache.org/job/hadoop-trunk-win/433/

[Apr 9, 2018 3:47:22 PM] (sunilg) YARN-7574. Add support for Node Labels on 
Auto Created Leaf Queue
[Apr 9, 2018 4:16:48 PM] (inigoiri) HDFS-13388. RequestHedgingProxyProvider 
calls multiple configured NNs
[Apr 9, 2018 5:09:25 PM] (inigoiri) HDFS-13380. RBF: mv/rm fail after the 
directory exceeded the quota
[Apr 9, 2018 5:48:42 PM] (james.clampffer) HDFS-13376. Specify minimum GCC 
version to avoid TLS support error in
[Apr 9, 2018 10:19:21 PM] (jlowe) YARN-7667. Docker Stop grace period should be 
configurable. Contributed
[Apr 9, 2018 11:37:49 PM] (Bharat) HADOOP-15328. Fix the typo in 
HttpAuthentication.md. Contributed by fang
[Apr 10, 2018 6:29:28 AM] (yqlin) HDFS-13410. RBF: Support federation with no 
subclusters. Contributed by
[Apr 10, 2018 7:34:42 AM] (yqlin) HDFS-13384. RBF: Improve timeout RPC call 
mechanism. Contributed by
[Apr 10, 2018 9:59:40 AM] (aajisaka) HADOOP-15374. Add links of the new 
features of 3.1.0 to the top page
[Apr 10, 2018 10:39:09 AM] (sunilg) YARN-7804. [UI2] Refresh action on Grid 
view page should not be




-1 overall


The following subsystems voted -1:
compile mvninstall unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h 00m 00s)
unit


Specific tests:

Failed junit tests :

   hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec 
   hadoop.fs.contract.rawlocal.TestRawlocalContractAppend 
   hadoop.fs.TestFileUtil 
   hadoop.fs.TestFsShellCopy 
   hadoop.fs.TestFsShellList 
   hadoop.fs.TestLocalFileSystem 
   hadoop.fs.TestRawLocalFileSystemContract 
   hadoop.fs.TestTrash 
   hadoop.fs.viewfs.TestViewFileSystemWithAuthorityLocalFileSystem 
   hadoop.http.TestHttpServer 
   hadoop.http.TestHttpServerLogs 
   hadoop.io.nativeio.TestNativeIO 
   hadoop.ipc.TestIPC 
   hadoop.ipc.TestSocketFactory 
   hadoop.metrics2.impl.TestStatsDMetrics 
   hadoop.metrics2.sink.TestRollingFileSystemSinkWithLocal 
   hadoop.security.TestSecurityUtil 
   hadoop.security.TestShellBasedUnixGroupsMapping 
   hadoop.security.token.TestDtUtilShell 
   hadoop.util.TestNativeCodeLoader 
   hadoop.fs.TestResolveHdfsSymlink 
   hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy 
   hadoop.hdfs.crypto.TestHdfsCryptoStreams 
   hadoop.hdfs.qjournal.client.TestQuorumJournalManager 
   hadoop.hdfs.qjournal.server.TestJournalNode 
   hadoop.hdfs.qjournal.server.TestJournalNodeSync 
   hadoop.hdfs.server.balancer.TestBalancer 
   hadoop.hdfs.server.balancer.TestBalancerRPCDelay 
   hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages 
   hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy 
   
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement 
   
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestProvidedImpl 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica 
   hadoop.hdfs.server.datanode.TestBlockPoolSliceStorage 
   hadoop.hdfs.server.datanode.TestBlockRecovery 
   hadoop.hdfs.server.datanode.TestBlockScanner 
   hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics 
   hadoop.hdfs.server.datanode.TestDataNodeFaultInjector 
   hadoop.hdfs.server.datanode.TestDataNodeMetrics 
   hadoop.hdfs.server.datanode.TestDataNodeUUID 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.server.datanode.TestDirectoryScanner 
   hadoop.hdfs.server.datanode.TestHSync 
   hadoop.hdfs.server.datanode.web.TestDatanodeHttpXFrame 
   hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand 
   hadoop.hdfs.server.diskbalancer.TestDiskBalancer 
   hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC 
   hadoop.hdfs.server.mover.TestMover 
   hadoop.hdfs.server.mover.TestStorageMover 
   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA 
   hadoop.hdfs.server.namenode.ha.TestDNFencing 
   hadoop.hdfs.server.namenode.ha.TestHAAppend 
   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA 
   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics 
   
hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot

[jira] [Created] (YARN-8143) Improve log message when Capacity Scheduler request allocation on node

2018-04-10 Thread Zian Chen (JIRA)

Zian Chen created YARN-8143:
---

 Summary: Improve log message when Capacity Scheduler request 
allocation on node
 Key: YARN-8143
 URL: https://issues.apache.org/jira/browse/YARN-8143
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: yarn
Reporter: Zian Chen
Assignee: Zian Chen


When scheduler request allocates container on the node with reserved containers 
on it, this log message will print very frequently which needs to be improved 
with more condition checks.
{code:java}
2018-02-02 11:41:13,105 INFO  capacity.CapacityScheduler 
(CapacityScheduler.java:tryCommit(2673)) - Allocation proposal accepted
2018-02-02 11:41:13,115 INFO  capacity.CapacityScheduler 
(CapacityScheduler.java:allocateContainerOnSingleNode(1391)) - Trying to 
fulfill reservation for application application_1517571510094_0003 on node: 
ctr-e137-1514896590304-52728-01-07.hwx.site:25454
2018-02-02 11:41:13,115 INFO  allocator.AbstractContainerAllocator 
(AbstractContainerAllocator.java:getCSAssignmentFromAllocateResult(97)) - 
Reserved container  application=application_1517571510094_0003 
resource= 
queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@3f04848e
 cluster=
{code}
 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-8142) yarn service application stops when AM is killed

2018-04-10 Thread Yesha Vora (JIRA)

Yesha Vora created YARN-8142:


 Summary: yarn service application stops when AM is killed
 Key: YARN-8142
 URL: https://issues.apache.org/jira/browse/YARN-8142
 Project: Hadoop YARN
  Issue Type: Bug
  Components: yarn-native-services
Reporter: Yesha Vora


Steps:

1) Launch sleeper job ( non-docker yarn service)

{code}

RUNNING: /usr/hdp/current/hadoop-yarn-client/bin/yarn app -launch 
fault-test-am-sleeper 
/usr/hdp/current/hadoop-yarn-client/yarn-service-examples/sleeper/sleeper.json

WARNING: YARN_LOG_DIR has been replaced by HADOOP_LOG_DIR. Using value of 
YARN_LOG_DIR.

WARNING: YARN_LOGFILE has been replaced by HADOOP_LOGFILE. Using value of 
YARN_LOGFILE.

WARNING: YARN_PID_DIR has been replaced by HADOOP_PID_DIR. Using value of 
YARN_PID_DIR.

WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.

18/04/06 22:24:24 WARN util.NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable

18/04/06 22:24:24 INFO client.AHSProxy: Connecting to Application History 
server at xxx:10200

18/04/06 22:24:24 INFO client.AHSProxy: Connecting to Application History 
server at xxx:10200

18/04/06 22:24:24 INFO client.ApiServiceClient: Loading service definition from 
local FS: 
/usr/hdp/current/hadoop-yarn-client/yarn-service-examples/sleeper/sleeper.json

18/04/06 22:24:26 INFO util.log: Logging initialized @3631ms

18/04/06 22:24:37 INFO client.ApiServiceClient: Application ID: 
application_1522887500374_0010

Exit Code: 0\{code}

2) Wait for sleeper component to be up

3) Kill AM process PID

 

Expected behavior:

New attempt of AM will be started. The pre-existing container will keep running

 

Actual behavior:

Application finishes with State : FINISHED and Final-State : ENDED

New attempt was never launched

Note: 

when the AM gets a SIGTERM and gracefully shuts itself down. It is shutting the 
entire app down instead of letting it continue to run for another attempt

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-8141) YARN Native Service: Respect YARN_CONTAINER_RUNTIME_DOCKER_LOCAL_RESOURCE_MOUNTS specified in service spec

2018-04-10 Thread Wangda Tan (JIRA)

Wangda Tan created YARN-8141:


 Summary: YARN Native Service: Respect 
YARN_CONTAINER_RUNTIME_DOCKER_LOCAL_RESOURCE_MOUNTS specified in service spec
 Key: YARN-8141
 URL: https://issues.apache.org/jira/browse/YARN-8141
 Project: Hadoop YARN
  Issue Type: Bug
  Components: yarn-native-services
Reporter: Wangda Tan


Existing YARN native service overwrites 
YARN_CONTAINER_RUNTIME_DOCKER_LOCAL_RESOURCE_MOUNTS regardless if user 
specified this in service spec or not. It is important to allow user to mount 
local folders like /etc/passwd, etc.

Following logic overwrites the 
YARN_CONTAINER_RUNTIME_DOCKER_LOCAL_RESOURCE_MOUNTS environment:
{code:java}
StringBuilder sb = new StringBuilder();
for (Entry mount : mountPaths.entrySet()) {
  if (sb.length() > 0) {
sb.append(",");
  }
  sb.append(mount.getKey());
  sb.append(":");
  sb.append(mount.getValue());
}
env.put("YARN_CONTAINER_RUNTIME_DOCKER_LOCAL_RESOURCE_MOUNTS", 
sb.toString());{code}
Inside AbstractLauncher.java



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-8140) Improve log message when launch cmd is ran for stopped yarn service

2018-04-10 Thread Yesha Vora (JIRA)

Yesha Vora created YARN-8140:


 Summary: Improve log message when launch cmd is ran for stopped 
yarn service
 Key: YARN-8140
 URL: https://issues.apache.org/jira/browse/YARN-8140
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: yarn-native-services
Affects Versions: 3.1.0
Reporter: Yesha Vora


Steps:

 1) Launch sleeper app

{code}

RUNNING: /usr/hdp/current/hadoop-yarn-client/bin/yarn app -launch 
sleeper2-duplicate-app-stopped 
/usr/hdp/3.0.0.0-xxx/hadoop-yarn/yarn-service-examples/sleeper/sleeper.json

WARNING: YARN_LOG_DIR has been replaced by HADOOP_LOG_DIR. Using value of 
YARN_LOG_DIR.

WARNING: YARN_LOGFILE has been replaced by HADOOP_LOGFILE. Using value of 
YARN_LOGFILE.

WARNING: YARN_PID_DIR has been replaced by HADOOP_PID_DIR. Using value of 
YARN_PID_DIR.

WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.

18/04/10 21:31:01 WARN util.NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable

18/04/10 21:31:01 INFO client.AHSProxy: Connecting to Application History 
server at xx:10200

18/04/10 21:31:01 INFO client.AHSProxy: Connecting to Application History 
server at xx:10200

18/04/10 21:31:01 INFO client.ApiServiceClient: Loading service definition from 
local FS: 
/usr/hdp/3.0.0.0-xxx/hadoop-yarn/yarn-service-examples/sleeper/sleeper.json

18/04/10 21:31:03 INFO util.log: Logging initialized @2818ms

18/04/10 21:31:10 INFO client.ApiServiceClient: Application ID: 
application_1523387473707_0007

Exit Code: 0\{code}

2) Stop the application

{code}

RUNNING: /usr/hdp/current/hadoop-yarn-client/bin/yarn app -stop 
sleeper2-duplicate-app-stopped

WARNING: YARN_LOG_DIR has been replaced by HADOOP_LOG_DIR. Using value of 
YARN_LOG_DIR.

WARNING: YARN_LOGFILE has been replaced by HADOOP_LOGFILE. Using value of 
YARN_LOGFILE.

WARNING: YARN_PID_DIR has been replaced by HADOOP_PID_DIR. Using value of 
YARN_PID_DIR.

WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.

18/04/10 21:31:14 WARN util.NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable

18/04/10 21:31:15 INFO client.AHSProxy: Connecting to Application History 
server at xx:10200

18/04/10 21:31:15 INFO client.AHSProxy: Connecting to Application History 
server at xx:10200

18/04/10 21:31:16 INFO util.log: Logging initialized @3034ms

18/04/10 21:31:17 INFO client.ApiServiceClient: Successfully stopped service 
sleeper2-duplicate-app-stopped

Exit Code: 0\{code}

3) Launch the application with same name

{code}

RUNNING: /usr/hdp/current/hadoop-yarn-client/bin/yarn app -launch 
sleeper2-duplicate-app-stopped 
/usr/hdp/3.0.0.0-xxx/hadoop-yarn/yarn-service-examples/sleeper/sleeper.json

WARNING: YARN_LOG_DIR has been replaced by HADOOP_LOG_DIR. Using value of 
YARN_LOG_DIR.

WARNING: YARN_LOGFILE has been replaced by HADOOP_LOGFILE. Using value of 
YARN_LOGFILE.

WARNING: YARN_PID_DIR has been replaced by HADOOP_PID_DIR. Using value of 
YARN_PID_DIR.

WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.

18/04/10 21:31:19 WARN util.NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable

18/04/10 21:31:19 INFO client.AHSProxy: Connecting to Application History 
server at xx:10200

18/04/10 21:31:19 INFO client.AHSProxy: Connecting to Application History 
server at xx:10200

18/04/10 21:31:19 INFO client.ApiServiceClient: Loading service definition from 
local FS: 
/usr/hdp/3.0.0.0-xxx/hadoop-yarn/yarn-service-examples/sleeper/sleeper.json

18/04/10 21:31:22 INFO util.log: Logging initialized @4456ms

18/04/10 21:31:22 ERROR client.ApiServiceClient: Service Instance dir already 
exists: 
hdfs://mycluster/user/hrt_qa/.yarn/services/sleeper2-duplicate-app-stopped/sleeper2-duplicate-app-stopped.json

Exit Code: 56

{code}

 

Here, launch cmd fails with "Service Instance dir already exists: 
hdfs://mycluster/user/hrt_qa/.yarn/services/sleeper2-duplicate-app-stopped/sleeper2-duplicate-app-stopped.json".

 

The log message should be more meaningful. It should return that 
"sleeper2-duplicate-app-stopped is in stopped state".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-7968) Reset the queue name in submission context while recovering an application

2018-04-10 Thread Yufei Gu (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu resolved YARN-7968.

Resolution: Won't Fix

> Reset the queue name in submission context while recovering an application
> --
>
> Key: YARN-7968
> URL: https://issues.apache.org/jira/browse/YARN-7968
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>Priority: Major
>
> After YARN-7139, the new application can get correct queue name in its 
> submission context. We need to do the same thing for application recovering. 
> {code}
>   if (isAppRecovering) {
> if (LOG.isDebugEnabled()) {
>   LOG.debug(applicationId
>   + " is recovering. Skip notifying APP_ACCEPTED");
> }
>   } else {
> // During tests we do not always have an application object, handle
> // it here but we probably should fix the tests
> if (rmApp != null && rmApp.getApplicationSubmissionContext() != null) 
> {
>   // Before we send out the event that the app is accepted is
>   // to set the queue in the submissionContext (needed on restore etc)
>   rmApp.getApplicationSubmissionContext().setQueue(queue.getName());
> }
> rmContext.getDispatcher().getEventHandler().handle(
> new RMAppEvent(applicationId, RMAppEventType.APP_ACCEPTED));
>   }
> {code}
> We can do it by moving the 
> {{rmApp.getApplicationSubmissionContext().setQueue}} block out of the if-else 
> block. cc [~wilfreds].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

2018-04-10 Thread Wangda Tan

Thanks @Konstantin for reporting this issue, I will post comments on the
JIRA (HADOOP-15205)

- Wangda

On Tue, Apr 10, 2018 at 12:08 PM, Konstantin Shvachko 
wrote:

> A note to release managers. As discussed in
> https://issues.apache.org/jira/browse/HADOOP-15205
> We are producing release artifacts without sources jars. See e.g.
> https://repository.apache.org/content/repositories/releases/
> org/apache/hadoop/hadoop-common/3.1.0/
> I believe this has something to do with maven deployment stage,
> potentially maven-source-plugin.
> This is similar for all releases now, and I believe it should be fixed.
>
> Thanks,
> --Konstantin
>
> On Fri, Apr 6, 2018 at 2:01 PM, Wangda Tan  wrote:
>
>> Thanks guys for the additional votes! I just sent out announcement email.
>>
>> Best,
>> Wangda
>>
>> On Fri, Apr 6, 2018 at 2:32 AM, 俊平堵  wrote:
>>
>> > Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
>> > based on:
>> >
>> > - Verified signatures
>> >
>> > - Verified checksums for source and binary artifacts
>> >
>> > - Built from source
>> >
>> > - Deployed a single node cluster
>> >
>> > - Verified web UIs, include Namenode, RM, etc.
>> >
>> > * Tried shell commands of HDFS and YARN
>> >
>> > * Ran sample MR jobs, include PI, Sleep, Terasort, etc.
>> >
>> >
>> > Thanks,
>> >
>> >
>> > Junping
>> >
>> >
>> >
>> > Wangda Tan 于2018年3月30日 周五下午12:15写道：
>> >
>> >> Hi folks,
>> >>
>> >> Thanks to the many who helped with this release since Dec 2017 [1].
>> We've
>> >> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>> >>
>> >> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>> >>
>> >> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> >> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>> >>
>> >> The maven artifacts are available via repository.apache.org at
>> >> https://repository.apache.org/content/repositories/orgapache
>> hadoop-1090/
>> >> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>> >>
>> >> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> >> include the first class GPU/FPGA support on YARN, Native services,
>> Support
>> >> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> >> block replicas to be provided by an external storage system, etc.
>> >>
>> >> For 3.1.0 RC0 vote discussion, please see [3].
>> >>
>> >> We’d like to use this as a starting release for 3.1.x [1], depending on
>> >> how
>> >> it goes, get it stabilized and potentially use a 3.1.1 in several
>> weeks as
>> >> the stable release.
>> >>
>> >> We have done testing with a pseudo cluster:
>> >> - Ran distributed job.
>> >> - GPU scheduling/isolation.
>> >> - Placement constraints (intra-application anti-affinity) by using
>> >> distributed shell.
>> >>
>> >> My +1 to start.
>> >>
>> >> Best,
>> >> Wangda/Vinod
>> >>
>> >> [1]
>> >> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> >> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> >> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
>> (3.1.0)
>> >> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>> BY
>> >> fixVersion ASC
>> >> [3]
>> >> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> >> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>> >>
>> >
>>
>
>

Re: [VOTE] Release Apache Hadoop 3.0.2 (RC0)

2018-04-10 Thread Konstantin Shvachko

A note to release managers. As discussed in
https://issues.apache.org/jira/browse/HADOOP-15205
We are producing release artifacts without sources jars. See e.g.
https://repository.apache.org/content/repositories/releases/org/apache/hadoop/hadoop-common/3.0.1/
Based on what is staged on Nexus for 3.0.2 (RC0) I see the next release
will not have sources as well.
https://repository.apache.org/#stagingRepositories
I believe this has something to do with maven deployment stage, potentially
maven-source-plugin.
This is similar for all releases now, and I believe it should be fixed.

Thanks,
--Konstantin

On Tue, Apr 10, 2018 at 8:32 AM, Ajay Kumar 
wrote:

> Thanks Lie for working on this.
>
>- Downloaded src tarball and verified checksums
>- Built from src on mac with java 1.8.0_111
>- Built a pseudo distributed hdfs cluster
>- Run test mr jobs (pi, dfsio,wordcount
>- Verified basic hdfs operations
>- Basic validation for webui
>
> ** I checked maven artifacts and it seems source jars are not there
> (checked hadoop-hdfs , hadoop-client). Not sure if they are required for
> release.
>
>
> On 4/9/18, 4:19 PM, "Xiao Chen"  wrote:
>
> Thanks Eddy for the effort!
>
> +1 (binding)
>
>- Downloaded src tarball and verified checksums
>- Built from src
>- Started a pseudo distributed hdfs cluster
>- Verified basic hdfs operations work
>- Sanity checked logs / webui
>
> Best,
> -Xiao
>
>
> On Mon, Apr 9, 2018 at 11:28 AM, Eric Payne  invalid>
> wrote:
>
> > Thanks a lot for working to produce this release.
> >
> > +1 (binding)
> > Tested the following:
> > - built from source and installed on 6-node pseudo-cluster
> > - tested Capacity Scheduler FairOrderingPolicy and
> FifoOrderingPolicy to
> > determine that capacity was assigned as expected in each case
> > - tested user weights with FifoOrderingPolicy to ensure that weights
> were
> > assigned to users as expected.
> >
> > Eric Payne
> >
> >
> >
> >
> >
> >
> > On Friday, April 6, 2018, 1:17:10 PM CDT, Lei Xu 
> wrote:
> >
> >
> >
> >
> >
> > Hi, All
> >
> > I've created release candidate RC-0 for Apache Hadoop 3.0.2.
> >
> > Please note: this is an amendment for Apache Hadoop 3.0.1 release to
> > fix shaded jars in apache maven repository. The codebase of 3.0.2
> > release is the same as 3.0.1.  New bug fixes will be included in
> > Apache Hadoop 3.0.3 instead.
> >
> > The release page is:
> > https://cwiki.apache.org/confluence/display/HADOOP/
> Hadoop+3.0+Release
> >
> > New RC is available at: http://home.apache.org/~lei/
> hadoop-3.0.2-RC0/
> >
> > The git tag is release-3.0.2-RC0, and the latest commit is
> > 5c141f7c0f24c12cb8704a6ccc1ff8ec991f41ee
> >
> > The maven artifacts are available at
> > https://repository.apache.org/content/repositories/
> orgapachehadoop-1096/
> >
> > Please try the release, especially, *verify the maven artifacts*,
> and vote.
> >
> > The vote will run 5 days, ending 4/11/2018.
> >
> > Thanks for everyone who helped to spot the error and proposed fixes!
> >
> > 
> -
> > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: mapreduce-dev-help@hadoop.
> apache.org
> >
> >
> > 
> -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> >
>
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>

Re: [VOTE] Release Apache Hadoop 3.0.2 (RC0)

2018-04-10 Thread Lei Xu

Ajay, thanks for spotting this.

I am working on fix the deploy.

On Tue, Apr 10, 2018 at 8:32 AM, Ajay Kumar  wrote:
> Thanks Lie for working on this.
>
>- Downloaded src tarball and verified checksums
>- Built from src on mac with java 1.8.0_111
>- Built a pseudo distributed hdfs cluster
>- Run test mr jobs (pi, dfsio,wordcount
>- Verified basic hdfs operations
>- Basic validation for webui
>
> ** I checked maven artifacts and it seems source jars are not there (checked 
> hadoop-hdfs , hadoop-client). Not sure if they are required for release.
>
>
> On 4/9/18, 4:19 PM, "Xiao Chen"  wrote:
>
> Thanks Eddy for the effort!
>
> +1 (binding)
>
>- Downloaded src tarball and verified checksums
>- Built from src
>- Started a pseudo distributed hdfs cluster
>- Verified basic hdfs operations work
>- Sanity checked logs / webui
>
> Best,
> -Xiao
>
>
> On Mon, Apr 9, 2018 at 11:28 AM, Eric Payne 
> 
> wrote:
>
> > Thanks a lot for working to produce this release.
> >
> > +1 (binding)
> > Tested the following:
> > - built from source and installed on 6-node pseudo-cluster
> > - tested Capacity Scheduler FairOrderingPolicy and FifoOrderingPolicy to
> > determine that capacity was assigned as expected in each case
> > - tested user weights with FifoOrderingPolicy to ensure that weights 
> were
> > assigned to users as expected.
> >
> > Eric Payne
> >
> >
> >
> >
> >
> >
> > On Friday, April 6, 2018, 1:17:10 PM CDT, Lei Xu  
> wrote:
> >
> >
> >
> >
> >
> > Hi, All
> >
> > I've created release candidate RC-0 for Apache Hadoop 3.0.2.
> >
> > Please note: this is an amendment for Apache Hadoop 3.0.1 release to
> > fix shaded jars in apache maven repository. The codebase of 3.0.2
> > release is the same as 3.0.1.  New bug fixes will be included in
> > Apache Hadoop 3.0.3 instead.
> >
> > The release page is:
> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release
> >
> > New RC is available at: http://home.apache.org/~lei/hadoop-3.0.2-RC0/
> >
> > The git tag is release-3.0.2-RC0, and the latest commit is
> > 5c141f7c0f24c12cb8704a6ccc1ff8ec991f41ee
> >
> > The maven artifacts are available at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1096/
> >
> > Please try the release, especially, *verify the maven artifacts*, and 
> vote.
> >
> > The vote will run 5 days, ending 4/11/2018.
> >
> > Thanks for everyone who helped to spot the error and proposed fixes!
> >
> > -
> > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
> >
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> >
>
>



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

2018-04-10 Thread Konstantin Shvachko

A note to release managers. As discussed in
https://issues.apache.org/jira/browse/HADOOP-15205
We are producing release artifacts without sources jars. See e.g.
https://repository.apache.org/content/repositories/releases/org/apache/hadoop/hadoop-common/3.1.0/
I believe this has something to do with maven deployment stage, potentially
maven-source-plugin.
This is similar for all releases now, and I believe it should be fixed.

Thanks,
--Konstantin

On Fri, Apr 6, 2018 at 2:01 PM, Wangda Tan  wrote:

> Thanks guys for the additional votes! I just sent out announcement email.
>
> Best,
> Wangda
>
> On Fri, Apr 6, 2018 at 2:32 AM, 俊平堵  wrote:
>
> > Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
> > based on:
> >
> > - Verified signatures
> >
> > - Verified checksums for source and binary artifacts
> >
> > - Built from source
> >
> > - Deployed a single node cluster
> >
> > - Verified web UIs, include Namenode, RM, etc.
> >
> > * Tried shell commands of HDFS and YARN
> >
> > * Ran sample MR jobs, include PI, Sleep, Terasort, etc.
> >
> >
> > Thanks,
> >
> >
> > Junping
> >
> >
> >
> > Wangda Tan 于2018年3月30日 周五下午12:15写道：
> >
> >> Hi folks,
> >>
> >> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
> >> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >>
> >> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> >>
> >> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> >> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >>
> >> The maven artifacts are available via repository.apache.org at
> >> https://repository.apache.org/content/repositories/
> orgapachehadoop-1090/
> >> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >>
> >> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> >> include the first class GPU/FPGA support on YARN, Native services,
> Support
> >> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> >> block replicas to be provided by an external storage system, etc.
> >>
> >> For 3.1.0 RC0 vote discussion, please see [3].
> >>
> >> We’d like to use this as a starting release for 3.1.x [1], depending on
> >> how
> >> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
> >> the stable release.
> >>
> >> We have done testing with a pseudo cluster:
> >> - Ran distributed job.
> >> - GPU scheduling/isolation.
> >> - Placement constraints (intra-application anti-affinity) by using
> >> distributed shell.
> >>
> >> My +1 to start.
> >>
> >> Best,
> >> Wangda/Vinod
> >>
> >> [1]
> >> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> >> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> >> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> >> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
> >> fixVersion ASC
> >> [3]
> >> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> >> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
> >>
> >
>

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2018-04-10 Thread Apache Jenkins Server

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/747/

[Apr 9, 2018 3:47:22 PM] (sunilg) YARN-7574. Add support for Node Labels on 
Auto Created Leaf Queue
[Apr 9, 2018 4:16:48 PM] (inigoiri) HDFS-13388. RequestHedgingProxyProvider 
calls multiple configured NNs
[Apr 9, 2018 5:09:25 PM] (inigoiri) HDFS-13380. RBF: mv/rm fail after the 
directory exceeded the quota
[Apr 9, 2018 5:48:42 PM] (james.clampffer) HDFS-13376. Specify minimum GCC 
version to avoid TLS support error in
[Apr 9, 2018 10:19:21 PM] (jlowe) YARN-7667. Docker Stop grace period should be 
configurable. Contributed
[Apr 9, 2018 11:37:49 PM] (Bharat) HADOOP-15328. Fix the typo in 
HttpAuthentication.md. Contributed by fang




-1 overall


The following subsystems voted -1:
unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.server.TestDiskFailures 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/747/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/747/artifact/out/diff-compile-javac-root.txt
  [288K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/747/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/747/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/747/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/747/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/747/artifact/out/whitespace-eol.txt
  [9.4M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/747/artifact/out/whitespace-tabs.txt
  [1.1M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/747/artifact/out/xml.txt
  [4.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/747/artifact/out/diff-javadoc-javadoc-root.txt
  [760K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/747/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [300K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/747/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/747/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
  [84K]

Powered by Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-8139) Skip node hostname resolution when running SLS.

2018-04-10 Thread Abhishek Modi (JIRA)

Abhishek Modi created YARN-8139:
---

 Summary: Skip node hostname resolution when running SLS.
 Key: YARN-8139
 URL: https://issues.apache.org/jira/browse/YARN-8139
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Abhishek Modi
Assignee: Abhishek Modi


Currently depending on the time taken in resolution of hostname, metrics of SLS 
gets skewed. To avoid this, in this fix we are introducing a flag which can be 
used to disable hostname resolutions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-8138) No containers pre-empted from another queue when using node labels

2018-04-10 Thread Charan Hebri (JIRA)

Charan Hebri created YARN-8138:
--

 Summary: No containers pre-empted from another queue when using 
node labels
 Key: YARN-8138
 URL: https://issues.apache.org/jira/browse/YARN-8138
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Charan Hebri


There seems to be an issue with pre-emption when using node labels with queue 
priority.

Test configuration:
queue A (capacity=50, priority=1)
queue B (capacity=50, priority=2)
both have accessible-node-labels set to x
A.accessible-node-labels.x.capacity = 50
B.accessible-node-labels.x.capacity = 50
Along with this pre-emption related properties have been set.

Test steps:
 - Set NM memory = 6000MB and containerMemory = 750MB
 - Submit an application A1 to B, with am-container = container = 
(6000-750-1500), no. of containers = 2
 - Submit an application A2 to A, with am-container = 750, container = 1500, no 
of containers = (NUM_NM-1)
 - Kill application A1
 - Submit an application A3 to B with am-container=container=5000, no. of 
containers=3
 - Expectation is that containers are pre-empted from application A2 to A3 but 
there is no container pre-emption happening
Container pre-emption is stuck with the message in the RM log,
{noformat}
2018-02-02 11:41:36,974 INFO capacity.CapacityScheduler 
(CapacityScheduler.java:tryCommit(2673)) - Allocation proposal accepted
2018-02-02 11:41:36,984 INFO capacity.CapacityScheduler 
(CapacityScheduler.java:allocateContainerOnSingleNode(1391)) - Trying to 
fulfill reservation for application application_1517571510094_0003 on node: 
XX:25454
2018-02-02 11:41:36,984 INFO allocator.AbstractContainerAllocator 
(AbstractContainerAllocator.java:getCSAssignmentFromAllocateResult(97)) - 
Reserved container application=application_1517571510094_0003 
resource= 
queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@3f04848e
 cluster=
2018-02-02 11:41:36,984 INFO capacity.CapacityScheduler 
(CapacityScheduler.java:tryCommit(2673)) - Allocation proposal accepted
2018-02-02 11:41:36,984 INFO capacity.CapacityScheduler 
(CapacityScheduler.java:allocateContainerOnSingleNode(1391)) - Trying to 
fulfill reservation for application application_1517571510094_0003 on node: 
XX:25454
2018-02-02 11:41:36,984 INFO allocator.AbstractContainerAllocator 
(AbstractContainerAllocator.java:getCSAssignmentFromAllocateResult(97)) - 
Reserved container application=application_1517571510094_0003 
resource= 
queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@3f04848e
 cluster=
2018-02-02 11:41:36,984 INFO capacity.CapacityScheduler 
(CapacityScheduler.java:tryCommit(2673)) - Allocation proposal accepted
2018-02-02 11:41:36,994 INFO capacity.CapacityScheduler 
(CapacityScheduler.java:allocateContainerOnSingleNode(1391)) - Trying to 
fulfill reservation for application application_1517571510094_0003 on node: 
XX:25454
2018-02-02 11:41:36,995 INFO allocator.AbstractContainerAllocator 
(AbstractContainerAllocator.java:getCSAssignmentFromAllocateResult(97)) - 
Reserved container application=application_1517571510094_0003 
resource= 
queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@3f04848e
 cluster={noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64

[jira] [Created] (YARN-8143) Improve log message when Capacity Scheduler request allocation on node

[jira] [Created] (YARN-8142) yarn service application stops when AM is killed

[jira] [Created] (YARN-8141) YARN Native Service: Respect YARN_CONTAINER_RUNTIME_DOCKER_LOCAL_RESOURCE_MOUNTS specified in service spec

[jira] [Created] (YARN-8140) Improve log message when launch cmd is ran for stopped yarn service

[jira] [Resolved] (YARN-7968) Reset the queue name in submission context while recovering an application

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

Re: [VOTE] Release Apache Hadoop 3.0.2 (RC0)

Re: [VOTE] Release Apache Hadoop 3.0.2 (RC0)

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[jira] [Created] (YARN-8139) Skip node hostname resolution when running SLS.

[jira] [Created] (YARN-8138) No containers pre-empted from another queue when using node labels

13 matches

Site Navigation

Mail list logo

Footer information