date:20180404

Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64

2018-04-04 Thread Apache Jenkins Server

For more details, see https://builds.apache.org/job/hadoop-trunk-win/427/

[Apr 3, 2018 2:59:20 PM] (haibochen) YARN-8051. 
TestRMEmbeddedElector#testCallbackSynchronization is flaky.
[Apr 3, 2018 3:31:34 PM] (stevel) HADOOP-14758. S3GuardTool.prune to handle 
UnsupportedOperationException.
[Apr 3, 2018 5:01:00 PM] (szegedim) YARN-8035. Uncaught exception in 
ContainersMonitorImpl during relaunch
[Apr 4, 2018 3:52:35 AM] (sunilg) YARN-7764. Findbugs warning: 
Resource#getResources may expose internal
[Apr 4, 2018 4:06:24 AM] (wangda) YARN-8106. Update 
LogAggregationIndexedFileController to use readFull




-1 overall


The following subsystems voted -1:
compile mvninstall unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h 00m 00s)
unit


Specific tests:

Failed junit tests :

   hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec 
   hadoop.fs.contract.rawlocal.TestRawlocalContractAppend 
   hadoop.fs.TestFileUtil 
   hadoop.fs.TestFsShellCopy 
   hadoop.fs.TestFsShellList 
   hadoop.fs.TestLocalFileSystem 
   hadoop.fs.TestRawLocalFileSystemContract 
   hadoop.fs.TestSymlinkLocalFSFileContext 
   hadoop.fs.TestTrash 
   hadoop.http.TestHttpServer 
   hadoop.http.TestHttpServerLogs 
   hadoop.io.nativeio.TestNativeIO 
   hadoop.ipc.TestSocketFactory 
   hadoop.metrics2.impl.TestStatsDMetrics 
   hadoop.metrics2.sink.TestRollingFileSystemSinkWithLocal 
   hadoop.security.TestSecurityUtil 
   hadoop.security.TestShellBasedUnixGroupsMapping 
   hadoop.security.token.TestDtUtilShell 
   hadoop.util.TestNativeCodeLoader 
   hadoop.util.TestWinUtils 
   hadoop.fs.TestResolveHdfsSymlink 
   hadoop.hdfs.crypto.TestHdfsCryptoStreams 
   hadoop.hdfs.qjournal.client.TestQuorumJournalManager 
   hadoop.hdfs.qjournal.server.TestJournalNode 
   hadoop.hdfs.qjournal.server.TestJournalNodeSync 
   hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages 
   hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy 
   
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement 
   
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestProvidedImpl 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica 
   hadoop.hdfs.server.datanode.TestBlockPoolSliceStorage 
   hadoop.hdfs.server.datanode.TestBlockRecovery 
   hadoop.hdfs.server.datanode.TestBlockScanner 
   hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics 
   hadoop.hdfs.server.datanode.TestDataNodeFaultInjector 
   hadoop.hdfs.server.datanode.TestDataNodeMetrics 
   hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade 
   hadoop.hdfs.server.datanode.TestDataNodeUUID 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.server.datanode.TestDirectoryScanner 
   hadoop.hdfs.server.datanode.TestHSync 
   hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage 
   hadoop.hdfs.server.datanode.web.TestDatanodeHttpXFrame 
   hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand 
   hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC 
   hadoop.hdfs.server.mover.TestMover 
   hadoop.hdfs.server.mover.TestStorageMover 
   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA 
   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA 
   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics 
   
hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot 
   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot 
   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots 
   hadoop.hdfs.server.namenode.snapshot.TestSnapRootDescendantDiff 
   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport 
   hadoop.hdfs.server.namenode.TestAddBlock 
   hadoop.hdfs.server.namenode.TestAuditLoggerWithCommands 
   hadoop.hdfs.server.namenode.TestCheckpoint 
   hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate 
   hadoop.hdfs.server.namenode.TestEditLogRace 
   hadoop.hdfs.server.namenode.TestFileTruncate 
   hadoop.hdfs.server.namenode.TestFsck 
   hadoop.hdfs.server.namenode.TestFSImage 
   hadoop.hdfs.server.namenode.TestFSImageWithSnapshot

[jira] [Created] (HDFS-13401) Different behavior in mkdir when the target folder of mount table is uncreated

2018-04-04 Thread Jianfei Jiang (JIRA)

Jianfei Jiang created HDFS-13401:


 Summary: Different behavior in mkdir when the target folder of 
mount table is uncreated
 Key: HDFS-13401
 URL: https://issues.apache.org/jira/browse/HDFS-13401
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Jianfei Jiang
Assignee: Jianfei Jiang


In federation cases, if we have a config like below:


 fs.viewfs.mounttable.ClusterX.link./foo
 hdfs://nn2-clusterx.example.com:8020/footarget
 

When the folder hdfs://nn2-clusterx.example.com:8020/projects/footarget is not 
actually exist, the command [[hdfs dfs ls /]] can still see the folder, but the 
folder is not actually exist. then we try to use mkdir command to create the 
viewfs folder [[hdfs dfs mkdir /foo]], the return code is true, but the folder 
is not created. However, when we use [[hdfs dfs mkdir -p /foo/xxx]] which have 
a deeper location to create folder, the folder is successfully created. The 
results in these two cases may be ambigiuous.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13399) Make Client field AlignmentContext non-static.

2018-04-04 Thread Plamen Jeliazkov (JIRA)

Plamen Jeliazkov created HDFS-13399:
---

 Summary: Make Client field AlignmentContext non-static.
 Key: HDFS-13399
 URL: https://issues.apache.org/jira/browse/HDFS-13399
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: HDFS-12943
Reporter: Plamen Jeliazkov
Assignee: Plamen Jeliazkov


In HDFS-12977, DFSClient's constructor was altered to make use of a new static 
method in Client that allowed one to set an AlignmentContext. This work is to 
remove that static field and make each DFSClient pass it's AlignmentContext 
down to the proxy Call level.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

2018-04-04 Thread Apache Jenkins Server

For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/185/

[Apr 4, 2018 2:16:20 PM] (stevel) HADOOP-14651. Update okhttp version to 2.7.5. 
Contributed by Ray Chiang




-1 overall


The following subsystems voted -1:
docker


Powered by Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13400) WebHDFS append returned stream has incorrectly set position

2018-04-04 Thread Erik Krogen (JIRA)

Erik Krogen created HDFS-13400:
--

 Summary: WebHDFS append returned stream has incorrectly set 
position
 Key: HDFS-13400
 URL: https://issues.apache.org/jira/browse/HDFS-13400
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 3.0.1, 2.7.5, 2.8.3, 2.9.0
Reporter: Erik Krogen






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13395) Ozone: Plugins support in HDSL Datanode Service

2018-04-04 Thread Nanda kumar (JIRA)

Nanda kumar created HDFS-13395:
--

 Summary: Ozone: Plugins support in HDSL Datanode Service
 Key: HDFS-13395
 URL: https://issues.apache.org/jira/browse/HDFS-13395
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Reporter: Nanda kumar
Assignee: Nanda kumar


As part of Datanode, we start {{HdslDatanodeService}} if {{ozone}} is enabled. 
We need provision to load plugins like {{Ozone Rest Service}} as part of  
{{HdslDatanodeService}} start.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13396) start-dfs.sh and stop-dfs.sh has malformed command; doesn't use workers file

2018-04-04 Thread Jeff Hubbs (JIRA)

Jeff Hubbs created HDFS-13396:
-

 Summary: start-dfs.sh and stop-dfs.sh has malformed command; 
doesn't use workers file
 Key: HDFS-13396
 URL: https://issues.apache.org/jira/browse/HDFS-13396
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs
Affects Versions: 3.0.1, 3.0.0
 Environment: Hadoop 3.0.1 binary distribution on Gentoo Linux, Icedtea 
JRE
Reporter: Jeff Hubbs


In 3.0.1's start-dfs.sh, the command to start the datanodes reads as follows:
{code:java}
hadoop_uservar_su hdfs datanode "${HADOOP_HDFS_HOME}/bin/hdfs" \
--workers \
--config "${HADOOP_CONF_DIR}" \
--daemon start \
datanode ${dataStartOpt}
{code}
 This doesn't work; executing the script produces this:
{code:java}
hdfs@msba02a ~ $ $HADOOP_HOME/sbin/start-dfs.sh
Starting namenodes on [msba02a.bus.emory.ddns]
Starting datanodes
^/opt/hadoop/3: ssh: Could not resolve hostname 
^/opt/hadoop/3.0.1/etc/hadoop/workers: Name or service not known
pdsh@msba02a: ^/opt/hadoop/3: ssh exited with exit code 255
Starting secondary namenodes [msba02a]

{code}
It misinterprets the value of HADOOP_CONF_DIR as one of the names of a machine 
it is supposed to access.

The workaround I developed involves the --hostnames option like so, changing 
the one-name-per-line workers file into a comma-separated list:
{code:java}
hadoop_uservar_su hdfs datanode "${HADOOP_HDFS_HOME}/bin/hdfs" \
    --workers \
    --hostnames `sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/,/g' 
${HADOOP_CONF_DIR}/workers` \
    --config "${HADOOP_CONF_DIR}" \
    --daemon start \
    datanode ${dataStartOpt}

{code}
A similar change had to be made to stop-dfs.sh. I've verified that 
HADOOP_HDFS_HOME and HADOOP_CONF_DIR are set correctly within the script at the 
point where this command executes.

This problem also exists in start-dfs.sh/stop-dfs.sh in 3.0.0, although the 
original invocation differs slightly from 3.0.1.

In 3.0.1, I'm running into another problem with getting datanodes started (was 
fine in 3.0.0) but I couldn't hit that problem until I got past this one.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2018-04-04 Thread Apache Jenkins Server

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/

[Apr 3, 2018 5:48:26 AM] (xiao) HADOOP-15317. Improve NetworkTopology 
chooseRandom's loop.
[Apr 3, 2018 6:10:08 AM] (xiao) HADOOP-15355. TestCommonConfigurationFields is 
broken by HADOOP-15312.
[Apr 3, 2018 7:08:40 AM] (yqlin) HDFS-13364. RBF: Support NamenodeProtocol in 
the Router. Contributed by
[Apr 3, 2018 2:59:20 PM] (haibochen) YARN-8051. 
TestRMEmbeddedElector#testCallbackSynchronization is flaky.
[Apr 3, 2018 3:31:34 PM] (stevel) HADOOP-14758. S3GuardTool.prune to handle 
UnsupportedOperationException.
[Apr 3, 2018 5:01:00 PM] (szegedim) YARN-8035. Uncaught exception in 
ContainersMonitorImpl during relaunch




-1 overall


The following subsystems voted -1:
findbugs unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
   org.apache.hadoop.yarn.api.records.Resource.getResources() may expose 
internal representation by returning Resource.resources At Resource.java:by 
returning Resource.resources At Resource.java:[line 234] 

Failed junit tests :

   hadoop.hdfs.server.datanode.TestDataNodeUUID 
   hadoop.hdfs.TestDFSClientRetries 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.server.namenode.TestReencryptionWithKMS 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/diff-compile-javac-root.txt
  [288K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/whitespace-eol.txt
  [9.2M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/whitespace-tabs.txt
  [1.1M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/xml.txt
  [4.0K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/diff-javadoc-javadoc-root.txt
  [760K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [424K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [76K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/741/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
  [84K]

Powered by Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.9.1 Release Plan

2018-04-04 Thread Wei-Chiu Chuang

Sorry Sammi I was late to this thread.
Please considering incorporating HDFS-11915. Sync rbw dir on the first
hsync() to avoid file lost on power failure.
I thought it was already in 2.9.1 but turns out it didn't land. The cherry
pick to branch-2.9 is conflict free.

On Mon, Apr 2, 2018 at 4:34 AM, Chen, Sammi  wrote:

> Hi All,
>
> Today I have created branch-2.9.1 from branch-2.9 and started creating the
> RC0  based on branch-2.9.1.   But due to the corporate network conditions
> and my not full privileges on Hadoop,   it will take a while for RC0 to
> come out.
>
> If you have anything want to commit to branch-2.9.1,  please let me know.
>
> Also I will update fix version of all  2.9.1 JIRAs and moved all
> unresolved JIRA with target version = 2.9.1 to 2.9.2.
>
>
>
> Bests,
> Sammi Chen
>
> -Original Message-
> From: Chen, Sammi [mailto:sammi.c...@intel.com]
> Sent: Friday, March 30, 2018 3:55 PM
> To: hdfs-dev ; mapreduce-...@hadoop.apache.org;
> common-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: Apache Hadoop 2.9.1 Release Plan
>
> Hi All,
>
> We have 47 changes on 2.9 branch since last release on Nov. 2017.   There
> are 7 blockers, 5 critical issues and rest are normal bug fixes and feature
> improvements.
>
>
>
>
>
> Here are current tasks targeting for 2.9.1.  No critical and blockers so
> far.
>
> https://issues.apache.org/jira/issues/?jql=%22Target+
> Version%2Fs%22+%3D+2.9.1+AND+%28project+%3D+hadoop+OR+
> project+%3D+%22Hadoop+HDFS%22+OR+project+%3D+%22Hadoop+YARN%
> 22+OR+project+%3D+%22Hadoop+Map%2FReduce%22+OR+project+%
> 3D+%22Hadoop+Common%22%29+AND+status+%21%3D+resolved+ORDER+
> BY+priority+DESC
>
>
> I plan to cut the 2.9.1 branch today, and try to deliver the RC0  ASAP.
>  Please let me know if you have any objections or suggestions.
>
>
>
>
>
>
> Bests,
>
> Sammi
>
>
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>


-- 
A very happy Clouderan

[jira] [Resolved] (HDFS-13397) start-dfs.sh and hdfs --daemon start datanode say "ERROR: Cannot set priority of datanode process XXXX"

2018-04-04 Thread Jeff Hubbs (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-13397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Hubbs resolved HDFS-13397.
---
  Resolution: Invalid
Release Note: This fix apparently does not work in all cases, will withdraw 
and re-post after further investigation

> start-dfs.sh and hdfs --daemon start datanode say "ERROR: Cannot set priority 
> of datanode process "
> ---
>
> Key: HDFS-13397
> URL: https://issues.apache.org/jira/browse/HDFS-13397
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: hdfs
>Affects Versions: 3.0.1
>Reporter: Jeff Hubbs
>Priority: Major
>
> When executing
> {code:java}
> $HADOOP_HOME/bin/hdfs --daemon start datanode
> {code}
> as a regular user (e.g. "hdfs") you achieve fail saying
> {code:java}
> ERROR: Cannot set priority of datanode process 
> {code}
> where  is some PID.
> It turned out that this is because at least on Gentoo Linux (and I think this 
> is pretty well universal), by default a regular user process can't increase 
> the priority of itself or any of the user's other processes. To fix this, I 
> added these lines to /etc/security/limits.conf [NOTE: the users hdfs, yarn, 
> and mapred are in the group called hadoop on this system]:
> {code:java}
> @hadoop    hard    nice    -15
> @hadoop    hard    priority    -15
> {code}
> This change will need to be made on all datanodes.
> The need to enable [at minimum] the hdfs user to raise its processes' 
> priority needs to be added to the documentation. This is not a problem I 
> observed under 3.0.0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

2018-04-04 Thread Steve Loughran

that's "dangerously interesting". I think you are right, and I also think it'll 
just be the version files which get generated

anyway, +1 binding

* ran my new Hadoop-3 profile on spark (SPARK-23807), with the committer 
binding, then my downstream tests. All is well, provided you also have a spark 
hive JAR patched to accept hadoop 3 as a legitimate hadoop version. That's an 
ongoing issue in the Spark project. With that JAR on my CP my downstream tests 
were all happy (yesterday)

* today the staging files seem to be missing, at least maven is unable to find 
them even when I turn the spark snapshots-and-staging profile on. That'll be 
the maven dist process at play, nothing else

On 4 Apr 2018, at 04:13, Wangda Tan 
> wrote:

Hi Vinod / Arpit,

I checked following versions:
- 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:

Jars in maven repo [1] are *always* different from jars in the binary
tarball [2]: (I only checked hadoop-yarn-api-version.jar)

(Following numbers are sizes of the jar)
2.6.5:
- Jar in Maven: 1896185
- Jar in tarball: 1891485

2.7.5:
- Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
- Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
(Even size is same, md5 is different)

2.8.3:
- Jar in Maven: 2451433
- Jar in tarball: 2438975

2.9.0:
- Jar in Maven: 2791477
- Jar in tarball: 289

3.0.1:
- Jar in Maven: 2852604
- Jar in tarball: 2851373

I guess the differences come from our release process.

Thanks,
Wangda

[1] Maven jars are downloaded from
https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/
/hadoop-yarn-api-.jar
[2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/

On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli 
>
wrote:

We vote on the source code. The binaries are convenience artifacts.

This is what I would do - (a) Just replace both the maven jars as well as
the binaries to be consistent and correct. And then (b) Give a couple more
days for folks who tested on the binaries to reverify - I count one such
clear vote as of now.

Thanks
+Vinod

On Apr 3, 2018, at 3:30 PM, Wangda Tan 
> wrote:

HI Arpit,

I think it won't match if we do rebuild. It should be fine as far as
they're signed, correct? I don't see any policy doesn't allow this.

Thanks,
Wangda

On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal 
>
wrote:

Thanks Wangda, I see the shaded jars now.

Are the repo jars required to be the same as the binary release? They
don’t match right now, probably they got rebuilt.

+1 (binding), modulo that remaining question.

* Verified signatures
* Verified checksums for source and binary artefacts
* Sanity checked jars on r.a.o.
* Built from source
* Deployed to 3 node secure cluster with NameNode HA
* Verified HDFS web UIs
* Tried out HDFS shell commands
* Ran sample MapReduce jobs

Thanks!

--
From: Wangda Tan >
Date: Monday, April 2, 2018 at 9:25 PM
To: Arpit Agarwal >
Cc: Gera Shegalov >, Sunil G 
>, "
yarn-...@hadoop.apache.org" 
>, Hdfs-dev <
hdfs-dev@hadoop.apache.org>, Hadoop Common 
>,
"mapreduce-...@hadoop.apache.org" 
>,
Vinod Kumar Vavilapalli >
Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

As pointed by Arpit, the previously deployed shared jars are incorrect.
Just redeployed jars and staged. @Arpit, could you please check the updated
Maven repo? https://repository.apache.org/content/repositories/
orgapachehadoop-1092

Since the jars inside binary tarballs are correct (
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
need roll another RC, just update Maven repo should be sufficient.

Best,
Wangda

On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan 
wrote:
Hi Arpit,

Thanks for pointing out this.

I just removed all .md5 files from artifacts. I found md5 checksums still
exist in .mds files and I didn't remove them from .mds file because it is
generated by create-release script and Apache guidance is "should not"
instead of "must not". Please let me know if you think they need to be
removed as well.

- Wangda

On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal mailto:aagar...@hortonworks.com>> wrote:
Thanks for

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

2018-04-04 Thread Suma Shivaprasad

+1 (non binding)


*Verified - User Group Queue mapping - Node labels with New UI- Dynamic
queuesThanksSuma*


On Wed, Apr 4, 2018 at 11:48 AM, Steve Loughran 
wrote:

> that's "dangerously interesting". I think you are right, and I also think
> it'll just be the version files which get generated
>
> anyway, +1 binding
>
> * ran my new Hadoop-3 profile on spark (SPARK-23807), with the committer
> binding, then my downstream tests. All is well, provided you also have a
> spark hive JAR patched to accept hadoop 3 as a legitimate hadoop version.
> That's an ongoing issue in the Spark project. With that JAR on my CP my
> downstream tests were all happy (yesterday)
>
> * today the staging files seem to be missing, at least maven is unable to
> find them even when I turn the spark snapshots-and-staging profile on.
> That'll be the maven dist process at play, nothing else
>
> On 4 Apr 2018, at 04:13, Wangda Tan  eele...@gmail.com>> wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 289
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/
> releases/content/org/apache/hadoop/hadoop-yarn-api/
> /hadoop-yarn-api-.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org>
> wrote:
>
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan  eele...@gmail.com>> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal  mailto:aagar...@hortonworks.com>>
> wrote:
>
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> --
> From: Wangda Tan >
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal  com>>
> Cc: Gera Shegalov >, Sunil G <
> sun...@apache.org>, "
> yarn-...@hadoop.apache.org" <
> yarn-...@hadoop.apache.org>, Hdfs-dev <
> hdfs-dev@hadoop.apache.org>, Hadoop
> Common  >>,
> "mapreduce-...@hadoop.apache.org"
> >,
> Vinod Kumar Vavilapalli >
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/repositories/
> orgapachehadoop-1092
>
> Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan 
> wrote:
> Hi Arpit,
>
> Thanks for

[jira] [Created] (HDFS-13397) start-dfs.sh and hdfs --daemon start datanode say "ERROR: Cannot set priority of datanode process XXXX"

2018-04-04 Thread Jeff Hubbs (JIRA)

Jeff Hubbs created HDFS-13397:
-

 Summary: start-dfs.sh and hdfs --daemon start datanode say "ERROR: 
Cannot set priority of datanode process "
 Key: HDFS-13397
 URL: https://issues.apache.org/jira/browse/HDFS-13397
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: hdfs
Affects Versions: 3.0.1
Reporter: Jeff Hubbs


When executing
{code:java}
$HADOOP_HOME/bin/hdfs --daemon start datanode
{code}
as a regular user (e.g. "hdfs") you achieve fail saying
{code:java}
ERROR: Cannot set priority of datanode process 
{code}
where  is some PID.

It turned out that this is because at least on Gentoo Linux (and I think this 
is pretty well universal), by default a regular user process can't increase the 
priority of itself or any of the user's other processes. To fix this, I added 
these lines to /etc/security/limits.conf [NOTE: the users hdfs, yarn, and 
mapred are in the group called hadoop on this system]:
{code:java}
@hadoop    hard    nice    -15
@hadoop    hard    priority    -15
{code}
This change will need to be made on all datanodes.

The need to enable [at minimum] the hdfs user to raise its processes' priority 
needs to be added to the documentation. This is not a problem I observed under 
3.0.0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.1 (RC1)

2018-04-04 Thread Lei Xu

Hi, All

Thanks Arpit to find that 3.0.1 artifacts missed shaded jars.  I will
create a new 3.0.2 release with the same codebase as 3.0.1, but deploy
it as shaded jars. A new vote thread will start later this week.

After that, 3.0.3 release will be cut from branch-3.0 and includes the
new bug fixes. Please change the target version of new fixes to 3.0.3.
I will also go over JIRAs to fix targeted versions for committed changes.

Best,

On Mon, Apr 2, 2018 at 6:08 PM, Lei Xu  wrote:
> Hi, Arpit
>
> I followed this instruction
> https://wiki.apache.org/hadoop/HowToRelease
>
> It instructs to use "mvn deploy -Psign -DskipTests -DskipShade".
>
> It seems wrong, given that 3.0.0 is a shaded jar.
> I will do another deploy to fix it.
>
>
> On Mon, Apr 2, 2018 at 5:31 PM, Arpit Agarwal  
> wrote:
>> Hi Lei,
>>
>> It looks like the release artefacts have dummy shaded jars. E.g.
>>
>> Repository Path:  
>> /org/apache/hadoop/hadoop-client-runtime/3.0.1/hadoop-client-runtime-3.0.1.jar
>> Uploaded by:  lei
>> Size: 44.47 KB
>> Uploaded Date:Fri Mar 16 2018 15:50:42 GMT-0700 (PDT)
>> Last Modified:Fri Mar 16 2018 15:50:42 GMT-0700 (PDT)
>>
>> https://repository.apache.org/index.html#view-repositories;releases~browsestorage~/org/apache/hadoop/hadoop-client-runtime/3.0.1/hadoop-client-runtime-3.0.1.jar
>>
>> Am I looking at this wrong or is this supposed to be the shaded jar which is 
>> ~20MB?
>>
>> Thanks,
>> Arpit
>>
>>
>>
>> On 3/23/18, 10:18 AM, "Lei Xu"  wrote:
>>
>> Hi, All
>>
>> Thanks everyone for voting! The vote passes successfully with 6
>> binding +1s, 7 non-binding +1s and no -1s.
>>
>> I will work on the staging and releases.
>>
>> Best,
>>
>>
>> On Fri, Mar 23, 2018 at 5:10 AM, Kuhu Shukla  
>> wrote:
>> > +1 (non-binding)
>> >
>> > Built from source.
>> > Installed on a pseudo distributed cluster.
>> > Ran word count job and basic hdfs commands.
>> >
>> > Thank you for the effort on this release.
>> >
>> > Regards,
>> > Kuhu
>> >
>> > On Thu, Mar 22, 2018 at 5:25 PM, Elek, Marton  wrote:
>> >
>> >>
>> >> +1 (non binding)
>> >>
>> >> I did a full build from source code, created a docker container and 
>> did
>> >> various basic level tests with robotframework based automation and
>> >> docker-compose based pseudo clusters[1].
>> >>
>> >> Including:
>> >>
>> >> * Hdfs federation smoke test
>> >> * Basic ViewFS configuration
>> >> * Yarn example jobs
>> >> * Spark example jobs (with and without yarn)
>> >> * Simple hive table creation
>> >>
>> >> Marton
>> >>
>> >>
>> >> [1]: https://github.com/flokkr/runtime-compose
>> >>
>> >> On 03/18/2018 05:11 AM, Lei Xu wrote:
>> >>
>> >>> Hi, all
>> >>>
>> >>> I've created release candidate RC-1 for Apache Hadoop 3.0.1
>> >>>
>> >>> Apache Hadoop 3.0.1 will be the first bug fix release for Apache
>> >>> Hadoop 3.0 release. It includes 49 bug fixes and security fixes, 
>> which
>> >>> include 12
>> >>> blockers and 17 are critical.
>> >>>
>> >>> Please note:
>> >>> * HDFS-12990. Change default NameNode RPC port back to 8020. It makes
>> >>> incompatible changes to Hadoop 3.0.0.  After 3.0.1 releases, Apache
>> >>> Hadoop 3.0.0 will be deprecated due to this change.
>> >>>
>> >>> The release page is:
>> >>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release
>> >>>
>> >>> New RC is available at: http://home.apache.org/~lei/hadoop-3.0.1-RC1/
>> >>>
>> >>> The git tag is release-3.0.1-RC1, and the latest commit is
>> >>> 496dc57cc2e4f4da117f7a8e3840aaeac0c1d2d0
>> >>>
>> >>> The maven artifacts are available at:
>> >>> 
>> https://repository.apache.org/content/repositories/orgapachehadoop-1081/
>> >>>
>> >>> Please try the release and vote; the vote will run for the usual 5
>> >>> days, ending on 3/22/2017 6pm PST time.
>> >>>
>> >>> Thanks!
>> >>>
>> >>> -
>> >>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> >>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>> >>>
>> >>>
>> >> -
>> >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>> >>
>> >>
>>
>>
>>
>> --
>> Lei (Eddy) Xu
>> Software Engineer, Cloudera
>>
>> -
>> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>>
>>
>>

[jira] [Created] (HDFS-13398) Hdfs recursive listing operation is very slow

2018-04-04 Thread Ajay Sachdev (JIRA)

Ajay Sachdev created HDFS-13398:
---

 Summary: Hdfs recursive listing operation is very slow
 Key: HDFS-13398
 URL: https://issues.apache.org/jira/browse/HDFS-13398
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs
Affects Versions: 2.7.1
 Environment: HCFS file system where HDP 2.6.1 is connected to ECS 
(Object Store).
Reporter: Ajay Sachdev
 Fix For: 2.7.1


The hdfs dfs -ls -R command is sequential in nature and is very slow for a HCFS 
system. We have seen around 6 mins for 40K directory/files structure.

The proposal is to use multithreading approach to speed up recursive list, du 
and count operations.

We have tried a ForkJoinPool implementation to improve performance for 
recursive listing operation.

[https://github.com/jasoncwik/hadoop-release/tree/parallel-fs-cli]

commit id : 

82387c8cd76c2e2761bd7f651122f83d45ae8876

Another implementation is to use Java Executor Service to improve performance 
to run listing operation in multiple threads in parallel. This has 
significantly reduced the time to 40 secs from 6 mins.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13394) Ozone: ContainerID has incorrect package name

2018-04-04 Thread Nanda kumar (JIRA)

Nanda kumar created HDFS-13394:
--

 Summary: Ozone: ContainerID has incorrect package name
 Key: HDFS-13394
 URL: https://issues.apache.org/jira/browse/HDFS-13394
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Reporter: Nanda kumar


{{ContainerID}} package name and the directory structure where the class is 
present doesn't match.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64

[jira] [Created] (HDFS-13401) Different behavior in mkdir when the target folder of mount table is uncreated

[jira] [Created] (HDFS-13399) Make Client field AlignmentContext non-static.

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[jira] [Created] (HDFS-13400) WebHDFS append returned stream has incorrectly set position

[jira] [Created] (HDFS-13395) Ozone: Plugins support in HDSL Datanode Service

[jira] [Created] (HDFS-13396) start-dfs.sh and stop-dfs.sh has malformed command; doesn't use workers file

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

Re: Apache Hadoop 2.9.1 Release Plan

[jira] [Resolved] (HDFS-13397) start-dfs.sh and hdfs --daemon start datanode say "ERROR: Cannot set priority of datanode process XXXX"

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[jira] [Created] (HDFS-13397) start-dfs.sh and hdfs --daemon start datanode say "ERROR: Cannot set priority of datanode process XXXX"

Re: [VOTE] Release Apache Hadoop 3.0.1 (RC1)

[jira] [Created] (HDFS-13398) Hdfs recursive listing operation is very slow

[jira] [Created] (HDFS-13394) Ozone: ContainerID has incorrect package name

16 matches

Site Navigation

Mail list logo

Footer information