[jira] [Created] (HADOOP-15380) TestViewFileSystemLocalFileSystem#testTrashRoot leaves an unnecessary file

[Andras Bokor (JIRA)]

Andras Bokor created HADOOP-15380:
-

 Summary: TestViewFileSystemLocalFileSystem#testTrashRoot leaves an 
unnecessary file
 Key: HADOOP-15380
 URL: https://issues.apache.org/jira/browse/HADOOP-15380
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Andras Bokor
Assignee: Andras Bokor


After running

{code}mvn test -Dtest=TestViewFileSystemLocalFileSystem#testTrashRoot
git status{code}
Git reports an untracked file: 
{{hadoop-common-project/hadoop-common/.debug.log.crc}}
It seems some cleanup issue.
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15379) Make IrqHandler.bind() public

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-15379:
---

 Summary: Make IrqHandler.bind() public
 Key: HADOOP-15379
 URL: https://issues.apache.org/jira/browse/HADOOP-15379
 Project: Hadoop Common
  Issue Type: Bug
  Components: util
Affects Versions: 3.1.0
Reporter: Steve Loughran


{{org.apache.hadoop.service.launcher.IrqHandler.bind()}} is package private

this means you can create an {{Interrupted}} handler in a different package, 
but you can't bind it to a signal.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/748/

[Apr 10, 2018 6:29:28 AM] (yqlin) HDFS-13410. RBF: Support federation with no 
subclusters. Contributed by
[Apr 10, 2018 7:34:42 AM] (yqlin) HDFS-13384. RBF: Improve timeout RPC call 
mechanism. Contributed by
[Apr 10, 2018 9:59:40 AM] (aajisaka) HADOOP-15374. Add links of the new 
features of 3.1.0 to the top page
[Apr 10, 2018 10:39:09 AM] (sunilg) YARN-7804. [UI2] Refresh action on Grid 
view page should not be
[Apr 10, 2018 2:10:44 PM] (wwei) HDFS-13420. License header is displayed in 
ArchivalStorage/MemoryStorage
[Apr 10, 2018 2:58:26 PM] (aajisaka) HADOOP-15376. Remove double semi colons on 
imports that make Clover fall
[Apr 10, 2018 6:05:00 PM] (rakeshr) HDFS-13328. Abstract ReencryptionHandler 
recursive logic in separate
[Apr 10, 2018 6:19:48 PM] (xiao) HDFS-13363. Record file path when FSDirAclOp 
throws AclException.
[Apr 10, 2018 6:42:54 PM] (xyao) HADOOP-15340. Provide meaningful RPC server 
name for RpcMetrics.
[Apr 10, 2018 9:40:49 PM] (eyang) YARN-7984. Improved YARN service stop/destroy 
and clean up.   
[Apr 10, 2018 9:44:55 PM] (jlowe) HADOOP-15357. 
Configuration.getPropsWithPrefix no longer does variable
[Apr 10, 2018 10:38:25 PM] (xiao) HADOOP-14445. Delegation tokens are not 
shared between KMS instances.
[Apr 10, 2018 11:25:00 PM] (eyang) YARN-7973. Added ContainerRelaunch feature 
for Docker containers.   
[Apr 11, 2018 12:32:38 AM] (wangda) YARN-8116. Nodemanager fails with 
NumberFormatException: For input
[Apr 11, 2018 12:33:15 AM] (wangda) YARN-8133. Doc link broken for yarn-service 
from overview page. (Rohith




-1 overall


The following subsystems voted -1:
unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.TestEncryptionZonesWithKMS 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.mapreduce.v2.TestSpeculativeExecution 
   hadoop.mapreduce.TestMapReduceLazyOutput 
   hadoop.mapreduce.v2.TestSpeculativeExecutionWithMRApp 
   hadoop.mapreduce.TestChild 
   hadoop.mapreduce.v2.TestMRJobs 
   hadoop.mapreduce.v2.TestNonExistentJob 
   hadoop.mapreduce.v2.TestRMNMInfo 
   hadoop.mapreduce.v2.TestUberAM 
   hadoop.mapreduce.v2.TestMRJobsWithHistoryService 
   hadoop.mapreduce.v2.TestMRJobsWithProfiler 
   hadoop.mapreduce.TestMROutputFormat 
   hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities 
   hadoop.mapreduce.TestMapReduce 
   hadoop.mapreduce.TestLocalRunner 
   hadoop.conf.TestNoDefaultsJobConf 
   hadoop.mapreduce.v2.TestMRAppWithCombiner 
   hadoop.mapreduce.v2.TestMiniMRProxyUser 
   hadoop.mapreduce.security.TestMRCredentials 
   hadoop.mapreduce.security.TestBinaryTokenFile 
   hadoop.mapreduce.v2.TestMROldApiJobs 
   hadoop.streaming.TestSymLink 
   hadoop.mapred.gridmix.TestDistCacheEmulation 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/748/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/748/artifact/out/diff-compile-javac-root.txt
  [288K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/748/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/748/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/748/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/748/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/748/artifact/out/whitespace-eol.txt
  [9.4M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/748/artifact/out/whitespace-tabs.txt
  [1.1M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/748/artifact/out/xml.txt
  [4.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/748/artifact/out/diff-javadoc-javadoc-root.txt
  [760K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/748/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [420K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/748/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
  [260K]
   

Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64

[Apache Jenkins Server]

For more details, see https://builds.apache.org/job/hadoop-trunk-win/434/

[Apr 10, 2018 2:10:44 PM] (wwei) HDFS-13420. License header is displayed in 
ArchivalStorage/MemoryStorage
[Apr 10, 2018 2:58:26 PM] (aajisaka) HADOOP-15376. Remove double semi colons on 
imports that make Clover fall
[Apr 10, 2018 6:05:00 PM] (rakeshr) HDFS-13328. Abstract ReencryptionHandler 
recursive logic in separate
[Apr 10, 2018 6:19:48 PM] (xiao) HDFS-13363. Record file path when FSDirAclOp 
throws AclException.
[Apr 10, 2018 6:42:54 PM] (xyao) HADOOP-15340. Provide meaningful RPC server 
name for RpcMetrics.
[Apr 10, 2018 9:40:49 PM] (eyang) YARN-7984. Improved YARN service stop/destroy 
and clean up.   
[Apr 10, 2018 9:44:55 PM] (jlowe) HADOOP-15357. 
Configuration.getPropsWithPrefix no longer does variable
[Apr 10, 2018 10:38:25 PM] (xiao) HADOOP-14445. Delegation tokens are not 
shared between KMS instances.
[Apr 10, 2018 11:25:00 PM] (eyang) YARN-7973. Added ContainerRelaunch feature 
for Docker containers.   
[Apr 11, 2018 12:32:38 AM] (wangda) YARN-8116. Nodemanager fails with 
NumberFormatException: For input
[Apr 11, 2018 12:33:15 AM] (wangda) YARN-8133. Doc link broken for yarn-service 
from overview page. (Rohith
[Apr 11, 2018 3:48:50 AM] (rohithsharmaks) YARN-7941. Transitive dependencies 
for component are not resolved.
[Apr 11, 2018 4:15:39 AM] (rohithsharmaks) MAPREDUCE-7062. Update 
mapreduce.job.tags description for making use for
[Apr 11, 2018 4:31:48 AM] (xiao) HDFS-13056. Expose file-level composite CRCs 
in HDFS which are
[Apr 11, 2018 6:55:38 AM] (fabbri) HADOOP-15346 S3ARetryPolicy for 
400/BadArgument to be "fail".
[Apr 11, 2018 9:15:25 AM] (wwei) YARN-8127. Resource leak when async scheduling 
is enabled. Contributed




-1 overall


The following subsystems voted -1:
compile mvninstall unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


Specific tests:

Failed junit tests :

   hadoop.security.authentication.util.TestZKSignerSecretProvider 
   hadoop.crypto.key.kms.server.TestKMS 
   hadoop.crypto.key.kms.server.TestKMSWithZK 
   hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem 
   hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem 
   hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem 
   hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem 
   hadoop.fs.http.server.TestHttpFSServer 
   hadoop.fs.http.server.TestHttpFSServerNoACLs 
   hadoop.fs.http.server.TestHttpFSServerNoXAttrs 
   hadoop.fs.http.server.TestHttpFSServerWebServer 
   hadoop.lib.service.hadoop.TestFileSystemAccessService 
   hadoop.lib.service.security.TestGroupsService 
   hadoop.test.TestHFSTestCase 
   hadoop.hdfs.nfs.nfs3.TestClientAccessPrivilege 
   hadoop.hdfs.nfs.nfs3.TestExportsTable 
   hadoop.hdfs.nfs.nfs3.TestNfs3HttpServer 
   hadoop.hdfs.nfs.nfs3.TestReaddir 
   hadoop.hdfs.nfs.nfs3.TestRpcProgramNfs3 
   hadoop.hdfs.nfs.nfs3.TestViewfsWithNfs3 
   hadoop.hdfs.nfs.nfs3.TestWrites 
   hadoop.hdfs.nfs.TestMountd 
   hadoop.fs.contract.router.TestRouterHDFSContractAppend 
   hadoop.fs.contract.router.TestRouterHDFSContractConcat 
   hadoop.fs.contract.router.TestRouterHDFSContractCreate 
   hadoop.fs.contract.router.TestRouterHDFSContractDelete 
   hadoop.fs.contract.router.TestRouterHDFSContractGetFileStatus 
   hadoop.fs.contract.router.TestRouterHDFSContractMkdir 
   hadoop.fs.contract.router.TestRouterHDFSContractOpen 
   hadoop.fs.contract.router.TestRouterHDFSContractRename 
   hadoop.fs.contract.router.TestRouterHDFSContractRootDirectory 
   hadoop.fs.contract.router.TestRouterHDFSContractSeek 
   hadoop.fs.contract.router.TestRouterHDFSContractSetTimes 
   hadoop.fs.contract.router.web.TestRouterWebHDFSContractAppend 
   hadoop.yarn.client.api.impl.TestTimelineClientForATS1_5 
   
hadoop.yarn.logaggregation.filecontroller.ifile.TestLogAggregationIndexFileController
 
   
hadoop.yarn.logaggregation.filecontroller.TestLogAggregationFileControllerFactory
 
   hadoop.yarn.logaggregation.TestAggregatedLogFormat 
   hadoop.yarn.logaggregation.TestAggregatedLogsBlock 
   hadoop.yarn.nodelabels.TestFileSystemNodeLabelsStore 
   hadoop.yarn.util.TestFSDownload 
   hadoop.yarn.util.TestWindowsBasedProcessTree 
   hadoop.yarn.server.federation.utils.TestFederationRegistryClient 
   hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor 
   
hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch 
   
hadoop.yarn.server.nodemanager.containermanager.linux.privileged.TestPrivilegedOperationExecutor
 
   
hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestCGroupsHandlerImpl
 
   
hadoop.yarn.server.nodemanager.containermanager.linux.runtime.docker.TestDockerCommandExecutor
 
   

[jira] [Created] (HADOOP-15381) KmsAcls should be reloaded periodically.

[Rushabh S Shah (JIRA)]

Rushabh S Shah created HADOOP-15381:
---

 Summary: KmsAcls should be reloaded periodically.
 Key: HADOOP-15381
 URL: https://issues.apache.org/jira/browse/HADOOP-15381
 Project: Hadoop Common
  Issue Type: Bug
  Components: kms
Reporter: Rushabh S Shah
Assignee: Rushabh S Shah


Currently {{KMSACLs}} is getting reloaded depending on whether {{kms-acls}} 
file is modified or not.
Since the access control is specified by {{AcccessControlList}}, the member may 
contain {{unix-group}} or {{net-group}}.
If we add a new user to any group, technically the file is not modified but the 
membership of a group is modified, so we need to reload the {{kms-acls}} file.
There are couple of solutions for this.
1. short-term solution: Reload the file periodically (like lets say every 15 
minutes).
2. long-term solution: Implement a {{refreshUserGroupMapping}} like 
protocol/servlet so whenever we change membership we can invoke that protocol 
and we get the updates.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-12527) Upgrade Avro dependency to 1.7.7 or later

[Akira Ajisaka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka resolved HADOOP-12527.

Resolution: Duplicate

> Upgrade Avro dependency to 1.7.7 or later
> -
>
> Key: HADOOP-12527
> URL: https://issues.apache.org/jira/browse/HADOOP-12527
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 2.7.1
>Reporter: Jonathan Kelly
>Assignee: Sean Busbey
>Priority: Major
>
> Hadoop has depended upon Avro 1.7.4 for a couple of years now (see 
> HADOOP-9672), but Apache Spark depends upon what is currently the latest 
> version of Avro (1.7.7).
> This can cause issues if Spark is configured to include the full Hadoop 
> classpath, as the classpath would then contain both Avro 1.7.4 and 1.7.7, 
> with the 1.7.4 classes possibly winning depending on ordering. Here is an 
> example of this issue: 
> http://stackoverflow.com/questions/33159254/avro-error-on-aws-emr/33403111#33403111
> Would it be possible to upgrade Hadoop's Avro dependency to 1.7.7 now?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.9.1 Release Plan

[Akira Ajisaka]

Hi Sammi,

Thank you for driving 2.9.1 release.

I've committed HDFS-13427 (only documentation bug fix) to branch-2.9.1.
Sorry for doing this without reading this thread.

Regards,
Akira

On 2018/04/07 4:12, Wei-Chiu Chuang wrote:

Done.

Both HDFS-11915 and HDFS-13364 are in branch-2.9 and branch-2.9.1.

Thank you Sammi for driving 2.9.1 release

On Fri, Apr 6, 2018 at 10:14 AM, Wei-Chiu Chuang 
wrote:



1.
   Looks like:
2. HDFS-13347  breaks
compilation, which is later fixed byHDFS-13364
. But HDFS-13364 is
only available in 2.9.2 (branch-2.9)





Before I cherrypick HDFS-11915, I will cherry pick HDFS-13364 first
into branch-2.9.1.


On Fri, Apr 6, 2018 at 9:46 AM, Chen, Sammi  wrote:


Hi Wei-Chiu,

HDFS-11915 improves the data reliability when there is power failure.
I'm happy to include it in 2.9.1 release.

Would you help to cherry-pick it to both branch-2.9 and branch-2.9.1?


Thanks,
Sammi

-Original Message-
From: Wei-Chiu Chuang [mailto:weic...@cloudera.com]
Sent: Thursday, April 5, 2018 1:30 AM
To: Chen, Sammi 
Cc: hdfs-dev ; mapreduce-...@hadoop.apache.or
g; common-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.9.1 Release Plan

Sorry Sammi I was late to this thread.
Please considering incorporating HDFS-11915. Sync rbw dir on the first
hsync() to avoid file lost on power failure.
I thought it was already in 2.9.1 but turns out it didn't land. The
cherry pick to branch-2.9 is conflict free.

On Mon, Apr 2, 2018 at 4:34 AM, Chen, Sammi  wrote:


Hi All,

Today I have created branch-2.9.1 from branch-2.9 and started creating

the

RC0  based on branch-2.9.1.   But due to the corporate network

conditions

and my not full privileges on Hadoop,   it will take a while for RC0 to
come out.

If you have anything want to commit to branch-2.9.1,  please let me

know.


Also I will update fix version of all  2.9.1 JIRAs and moved all
unresolved JIRA with target version = 2.9.1 to 2.9.2.



Bests,
Sammi Chen

-Original Message-
From: Chen, Sammi [mailto:sammi.c...@intel.com]
Sent: Friday, March 30, 2018 3:55 PM
To: hdfs-dev ;
mapreduce-...@hadoop.apache.org; common-dev@hadoop.apache.org;
yarn-...@hadoop.apache.org
Subject: Apache Hadoop 2.9.1 Release Plan

Hi All,

We have 47 changes on 2.9 branch since last release on Nov. 2017.

  There

are 7 blockers, 5 critical issues and rest are normal bug fixes and
feature improvements.





Here are current tasks targeting for 2.9.1.  No critical and blockers
so far.

https://issues.apache.org/jira/issues/?jql=%22Target+
Version%2Fs%22+%3D+2.9.1+AND+%28project+%3D+hadoop+OR+
project+%3D+%22Hadoop+HDFS%22+OR+project+%3D+%22Hadoop+YARN%
22+OR+project+%3D+%22Hadoop+Map%2FReduce%22+OR+project+%
3D+%22Hadoop+Common%22%29+AND+status+%21%3D+resolved+ORDER+
BY+priority+DESC


I plan to cut the 2.9.1 branch today, and try to deliver the RC0  ASAP.
  Please let me know if you have any objections or suggestions.






Bests,

Sammi




-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org





--
A very happy Clouderan





--
A very happy Clouderan







-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15382) Log kinit output in credential renewal thread

[Wei-Chiu Chuang (JIRA)]

Wei-Chiu Chuang created HADOOP-15382:


 Summary: Log kinit output in credential renewal thread
 Key: HADOOP-15382
 URL: https://issues.apache.org/jira/browse/HADOOP-15382
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Reporter: Wei-Chiu Chuang


We currently run kinit command in a thread to renew kerberos credentials 
periodically. 
{code:java}
Shell.execCommand(cmd, "-R");
if (LOG.isDebugEnabled()) {
  LOG.debug("renewed ticket");
}
{code}
It seems useful to log the output of the kinit too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org