[jira] [Created] (HDFS-14117) RBF:We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled

[venkata ram kumar ch (JIRA)]

venkata ram kumar ch created HDFS-14117:
---

 Summary: RBF:We can only delete the files or dirs of one 
subcluster in a cluster with multiple subclusters when trash is enabled
 Key: HDFS-14117
 URL: https://issues.apache.org/jira/browse/HDFS-14117
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: venkata ram kumar ch


When we delete files or dirs in hdfs, it will move the deleted files or dirs to 
trash by default.

But in the global path we can only mount one trash dir /user. So we mount trash 
dir /user of the subcluster ns1 to the global path /user. Then we can delete 
files or dirs of ns1, but when we delete the files or dirs of another 
subcluser, such as hacluster, it will be failed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-13154) Webhdfs : update the Document for allow/disallow snapshots

[Brahma Reddy Battula (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-13154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula resolved HDFS-13154.
-
Resolution: Duplicate

Closing as duplicate to HDFS-13870, As it's already committed.

> Webhdfs : update the Document for allow/disallow snapshots
> --
>
> Key: HDFS-13154
> URL: https://issues.apache.org/jira/browse/HDFS-13154
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs, webhdfs
>Affects Versions: 2.8.2
>Reporter: Harshakiran Reddy
>Assignee: usharani
>Priority: Minor
>
> There is no Document for Allow/Disallow snapshots.
> http://hadoop.apache.org/docs/r2.8.3/hadoop-project-dist/hadoop-hdfs/WebHDFS.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

[Sunil G]

Thanks @Eric Payne 
Due to another issue, we have to spin an RC1. In this case, could we revert
the patch which caused this problem.

Cancelling this RC0. Thanks to everyone who voted.
I will spin RC1 as soon as possible.

- Sunil


On Fri, Nov 30, 2018 at 4:14 AM Eric Payne 
wrote:

> The problem is not with preemption. The yarn-site.xml that I use for my
> pseudo-cluster includes a second xml:
> xi:include href=".../yarn-scheduler.xml"
>
> The property for yarn.resourcemanager.scheduler.monitor.enable = true is
> in this yarn-scheduler.xml.
>
> This value IS READ when then RM starts.
>
> However, when the refreshQueues command is run, this value IS NOT READ.
>
> So, it looks like xml include files are not read on refresh. This will
> affect any property. I just happened to notice it on the preemption
> properties.
>
> I would like input from all of you to determine if this is a blocker for
> release. I'm on the fence.
>
> Thanks,
> -Eric
>
>
>
>
>
>
> On Wednesday, November 28, 2018, 4:58:50 PM CST, Eric Payne <
> erichadoo...@yahoo.com.INVALID> wrote:
>
>
>
>
>
> Sunil,
>
> So, the basic symptoms are that if preemption is enabled on any queue, the
> preemption is disabled after a 'yarn rm -refreshQueues'. In addition, all
> of the preemption-specific properties are set back to the default values.
>
> This was introduced in branch-3.1, so it is NOT new behavior for release
> 3.2.0. I am still tracking down the cause. I will open a JIRA once I do
> further investigation if there is not one already.
>
> This will be a problem for installations which use preemption and which
> use the refreshQueues feature.
>
> Thanks,
> -Eric
>
>
> On Wednesday, November 28, 2018, 11:47:06 AM CST, Eric Payne <
> eric.payne1...@yahoo.com> wrote:
>
>
>
>
>
> Sunil, thanks for all of the hard work on this release.
>
> I have discovered that queue refresh doesn't work in some cases. For
> example, when I change
> yarn.scheduler.capacity.root.default.disable_preemption, it doesn't take
> effect unless I restart the RM.
>
> I am still investigating, but I thought I should bring this up asap.
>
> Thanks,
> -Eric
>
>
>
>
> On Friday, November 23, 2018, 6:07:04 AM CST, Sunil G 
> wrote:
>
>
>
>
>
> Hi folks,
>
>
>
> Thanks to all contributors who helped in this release [1]. I have created
>
> first release candidate (RC0) for Apache Hadoop 3.2.0.
>
>
> Artifacts for this RC are available here:
>
> http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/
>
>
>
> RC tag in git is release-3.2.0-RC0.
>
>
>
> The maven artifacts are available via repository.apache.org at
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1174/
>
>
> This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.
>
>
>
> 3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
> additions
>
> are the highlights of this release.
>
> 1. Node Attributes Support in YARN
>
> 2. Hadoop Submarine project for running Deep Learning workloads on YARN
>
> 3. Support service upgrade via YARN Service API and CLI
>
> 4. HDFS Storage Policy Satisfier
>
> 5. Support Windows Azure Storage - Blob file system in Hadoop
>
> 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
>
> 7. Improvements in Router-based HDFS federation
>
>
>
> Thanks to Wangda, Vinod, Marton for helping me in preparing the release.
>
> I have done few testing with my pseudo cluster. My +1 to start.
>
>
>
> Regards,
>
> Sunil
>
>
>
> [1]
>
>
> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
> AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
> ORDER BY fixVersion ASC
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (HDDS-885) Fix test failures due to ChecksumData

[Hanisha Koneru (JIRA)]

Hanisha Koneru created HDDS-885:
---

 Summary: Fix test failures due to ChecksumData
 Key: HDDS-885
 URL: https://issues.apache.org/jira/browse/HDDS-885
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Hanisha Koneru
Assignee: Hanisha Koneru


HDDS-284 introduced test failures in the following:

# TestHddsDispatcher
# TestHddsDispatcher
# TestOzoneConfigurationFields
# TestBlockDeletingService
# TestBlockData
# TestContainerSmallFile

This Jira aims to fix these.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-14084) Need for more stats in DFSClient

[Pranay Singh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-14084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pranay Singh resolved HDFS-14084.
-
Resolution: Fixed

> Need for more stats in DFSClient
> 
>
> Key: HDFS-14084
> URL: https://issues.apache.org/jira/browse/HDFS-14084
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Pranay Singh
>Assignee: Pranay Singh
>Priority: Minor
> Attachments: HDFS-14084.001.patch
>
>
> The usage of HDFS has changed from being used as a map-reduce filesystem, now 
> it's becoming more of like a general purpose filesystem. In most of the cases 
> there are issues with the Namenode so we have metrics to know the workload or 
> stress on Namenode.
> However, there is a need to have more statistics collected for different 
> operations/RPCs in DFSClient to know which RPC operations are taking longer 
> time or to know what is the frequency of the operation.These statistics can 
> be exposed to the users of DFS Client and they can periodically log or do 
> some sort of flow control if the response is slow. This will also help to 
> isolate HDFS issue in a mixed environment where on a node say we have Spark, 
> HBase and Impala running together. We can check the throughput of different 
> operation across client and isolate the problem caused because of noisy 
> neighbor or network congestion or shared JVM.
> We have dealt with several problems from the field for which there is no 
> conclusive evidence as to what caused the problem. If we had metrics or stats 
> in DFSClient we would be better equipped to solve such complex problems.
> List of jiras for reference:
> -
>  HADOOP-15538 HADOOP-15530 ( client side deadlock)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Reopened] (HDFS-14084) Need for more stats in DFSClient

[Pranay Singh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-14084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pranay Singh reopened HDFS-14084:
-

> Need for more stats in DFSClient
> 
>
> Key: HDFS-14084
> URL: https://issues.apache.org/jira/browse/HDFS-14084
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Pranay Singh
>Assignee: Pranay Singh
>Priority: Minor
> Attachments: HDFS-14084.001.patch
>
>
> The usage of HDFS has changed from being used as a map-reduce filesystem, now 
> it's becoming more of like a general purpose filesystem. In most of the cases 
> there are issues with the Namenode so we have metrics to know the workload or 
> stress on Namenode.
> However, there is a need to have more statistics collected for different 
> operations/RPCs in DFSClient to know which RPC operations are taking longer 
> time or to know what is the frequency of the operation.These statistics can 
> be exposed to the users of DFS Client and they can periodically log or do 
> some sort of flow control if the response is slow. This will also help to 
> isolate HDFS issue in a mixed environment where on a node say we have Spark, 
> HBase and Impala running together. We can check the throughput of different 
> operation across client and isolate the problem caused because of noisy 
> neighbor or network congestion or shared JVM.
> We have dealt with several problems from the field for which there is no 
> conclusive evidence as to what caused the problem. If we had metrics or stats 
> in DFSClient we would be better equipped to solve such complex problems.
> List of jiras for reference:
> -
>  HADOOP-15538 HADOOP-15530 ( client side deadlock)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

[Eric Payne]

The problem is not with preemption. The yarn-site.xml that I use for my 
pseudo-cluster includes a second xml:
xi:include href=".../yarn-scheduler.xml"

The property for yarn.resourcemanager.scheduler.monitor.enable = true is in 
this yarn-scheduler.xml.

This value IS READ when then RM starts.

However, when the refreshQueues command is run, this value IS NOT READ.

So, it looks like xml include files are not read on refresh. This will affect 
any property. I just happened to notice it on the preemption properties.

I would like input from all of you to determine if this is a blocker for 
release. I'm on the fence.

Thanks,
-Eric






On Wednesday, November 28, 2018, 4:58:50 PM CST, Eric Payne 
 wrote: 





Sunil,

So, the basic symptoms are that if preemption is enabled on any queue, the 
preemption is disabled after a 'yarn rm -refreshQueues'. In addition, all of 
the preemption-specific properties are set back to the default values.

This was introduced in branch-3.1, so it is NOT new behavior for release 3.2.0. 
I am still tracking down the cause. I will open a JIRA once I do further 
investigation if there is not one already.

This will be a problem for installations which use preemption and which use the 
refreshQueues feature.

Thanks,
-Eric


On Wednesday, November 28, 2018, 11:47:06 AM CST, Eric Payne 
 wrote: 





Sunil, thanks for all of the hard work on this release.

I have discovered that queue refresh doesn't work in some cases. For example, 
when I change yarn.scheduler.capacity.root.default.disable_preemption, it 
doesn't take effect unless I restart the RM.

I am still investigating, but I thought I should bring this up asap.

Thanks,
-Eric




On Friday, November 23, 2018, 6:07:04 AM CST, Sunil G  
wrote: 





Hi folks,



Thanks to all contributors who helped in this release [1]. I have created

first release candidate (RC0) for Apache Hadoop 3.2.0.


Artifacts for this RC are available here:

http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/



RC tag in git is release-3.2.0-RC0.



The maven artifacts are available via repository.apache.org at

https://repository.apache.org/content/repositories/orgapachehadoop-1174/


This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.



3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
additions

are the highlights of this release.

1. Node Attributes Support in YARN

2. Hadoop Submarine project for running Deep Learning workloads on YARN

3. Support service upgrade via YARN Service API and CLI

4. HDFS Storage Policy Satisfier

5. Support Windows Azure Storage - Blob file system in Hadoop

6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a

7. Improvements in Router-based HDFS federation



Thanks to Wangda, Vinod, Marton for helping me in preparing the release.

I have done few testing with my pseudo cluster. My +1 to start.



Regards,

Sunil



[1]

https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E

[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
ORDER BY fixVersion ASC

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-13547) Add ingress port based sasl resolver

[Chen Liang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-13547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chen Liang resolved HDFS-13547.
---
   Resolution: Fixed
Fix Version/s: 3.1.1

> Add ingress port based sasl resolver
> 
>
> Key: HDFS-13547
> URL: https://issues.apache.org/jira/browse/HDFS-13547
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: security
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
> Fix For: 3.2.0, 3.1.1
>
> Attachments: HDFS-13547.001.patch, HDFS-13547.002.patch, 
> HDFS-13547.003.patch, HDFS-13547.004.patch
>
>
> This Jira extends the SASL properties resolver interface to take an ingress 
> port parameter, and also adds an implementation based on this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-884) Fix merge issue that causes NPE OzoneManager#httpServer

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDDS-884:
---

 Summary: Fix merge issue that causes NPE OzoneManager#httpServer
 Key: HDDS-884
 URL: https://issues.apache.org/jira/browse/HDDS-884
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


There is one line to instantiate httpServer missing due to the code movement on 
HDDS-4. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-14116) Fix a potential class cast error in ObserverReadProxyProvider

[Chen Liang (JIRA)]

Chen Liang created HDFS-14116:
-

 Summary: Fix a potential class cast error in 
ObserverReadProxyProvider
 Key: HDFS-14116
 URL: https://issues.apache.org/jira/browse/HDFS-14116
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: hdfs-client
Reporter: Chen Liang


Currently in {{ObserverReadProxyProvider}} constructor there is this line 
{code}
((ClientHAProxyFactory) factory).setAlignmentContext(alignmentContext);
{code}
This could potentially cause failure, because it is possible that factory can 
not be casted here. Specifically,  
{{NameNodeProxiesClient.createFailoverProxyProvider}} is where the constructor 
will be called, and there are two paths that could call into this:
(1).{{NameNodeProxies.createProxy}}
(2).{{NameNodeProxiesClient.createFailoverProxyProvider}}

(2) works fine because it always uses {{ClientHAProxyFactory}} but (1) uses 
{{NameNodeHAProxyFactory}} which can not be casted to {{ClientHAProxyFactory}}, 
this happens when, for example, running NNThroughputBenmarck. To fix this we 
can at least:
1. introduce setAlignmentContext to HAProxyFactory which is the parent of both  
ClientHAProxyFactory and NameNodeHAProxyFactory OR
2. only setAlignmentContext when it is ClientHAProxyFactory by, say, having a 
if check with reflection. 
Depending on whether it make sense to have alignment context for the case (1) 
calling code paths.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Reopened] (HDFS-13547) Add ingress port based sasl resolver

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-13547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli reopened HDFS-13547:


Oh, and this never made it to branch-3 either. [~vagarychen], I am reopening 
this, please put this in branch-3 too.

> Add ingress port based sasl resolver
> 
>
> Key: HDFS-13547
> URL: https://issues.apache.org/jira/browse/HDFS-13547
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: security
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HDFS-13547.001.patch, HDFS-13547.002.patch, 
> HDFS-13547.003.patch, HDFS-13547.004.patch
>
>
> This Jira extends the SASL properties resolver interface to take an ingress 
> port parameter, and also adds an implementation based on this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDDS-883) Chunk writes should validate the checksum before overwriting the existing chunk file

[Shashikant Banerjee (JIRA)]

Shashikant Banerjee created HDDS-883:


 Summary: Chunk writes should validate the checksum before 
overwriting the existing chunk file
 Key: HDDS-883
 URL: https://issues.apache.org/jira/browse/HDDS-883
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
  Components: Ozone Datanode
Affects Versions: 0.4.0
Reporter: Shashikant Banerjee
Assignee: Shashikant Banerjee
 Fix For: 0.4.0


When writeChunk request arrives via Ratis, it may happen that the tmp chunk 
file already exist because of ratis retry on stateMachine timeout during 
writeStateMachineData phase. In such cases, we can just validate the length and 
checksum if present before blindly overwriting it and return. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-14115) TestNamenodeCapacityReport#testXceiverCount is flaky

[Kitti Nanasi (JIRA)]

Kitti Nanasi created HDFS-14115:
---

 Summary: TestNamenodeCapacityReport#testXceiverCount is flaky
 Key: HDFS-14115
 URL: https://issues.apache.org/jira/browse/HDFS-14115
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.1.1
Reporter: Kitti Nanasi
Assignee: Kitti Nanasi


TestNamenodeCapacityReport#testXceiverCount sometimes fails with the following 
error:

{code}
2018-11-28 17:33:45,816 INFO  DataNode - PacketResponder: 
BP-645736292-172.17.0.2-1543426416580:blk_1073741828_1004, 
type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=2:[127.0.0.1:37115, 
127.0.0.1:35107] terminating
2018-11-28 17:33:45,817 INFO  StateChange - DIR* completeFile: /f3 is closed by 
DFSClient_NONMAPREDUCE_1933849415_1
2018-11-28 17:33:45,817 INFO  ExitUtil - Exiting with status 1: Block report 
processor encountered fatal exception: java.lang.AssertionError: Negative 
replicas!
2018-11-28 17:33:45,818 ERROR ExitUtil - Terminate called
1: Block report processor encountered fatal exception: 
java.lang.AssertionError: Negative replicas!
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:304)
at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:4807)
Exception in thread "Block report processor" 1: Block report processor 
encountered fatal exception: java.lang.AssertionError: Negative replicas!
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:304)
at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:4807)

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/972/

[Nov 27, 2018 11:57:24 PM] (aengineer) HDDS-846. Exports ozone metrics to 
prometheus. Contributed by Elek,
[Nov 28, 2018 8:42:14 AM] (sumasai) YARN-9034. ApplicationCLI should have 
option to take clusterId.
[Nov 28, 2018 8:46:53 AM] (sumasai) YARN-9044. LogsCLI should contact ATSv2 for 
-am option. Contributed by
[Nov 28, 2018 8:57:42 AM] (aajisaka) MAPREDUCE-6190. If a task stucks before 
its first heartbeat, it never
[Nov 28, 2018 2:46:11 PM] (jlowe) YARN-8812. Containers fail during creating a 
symlink which started with
[Nov 28, 2018 5:45:09 PM] (mackrorysd) HADOOP-15370. S3A log message on rm 
s3a://bucket/ not intuitive.
[Nov 28, 2018 5:45:09 PM] (mackrorysd) HADOOP-15798. LocalMetadataStore put() 
does not retain isDeleted in
[Nov 28, 2018 5:45:09 PM] (mackrorysd) HADOOP-15947. Fix 
ITestDynamoDBMetadataStore test error issues.
[Nov 28, 2018 6:53:12 PM] (hanishakoneru) HDDS-284. Checksum for ChunksData.
[Nov 28, 2018 7:25:47 PM] (gifuma) HDFS-14108. Performance improvement in 
BlockManager Data Structures.
[Nov 28, 2018 7:33:22 PM] (gifuma) HDFS-14102. Performance improvement in 
BlockPlacementPolicyDefault.
[Nov 28, 2018 8:54:59 PM] (jlowe) MAPREDUCE-7164. FileOutputCommitter does not 
report progress while
[Nov 28, 2018 9:36:21 PM] (wangda) YARN-9030. Log aggregation changes to handle 
filesystems which do not
[Nov 28, 2018 10:09:52 PM] (wangda) YARN-8882. [YARN-8851] Add a shared device 
mapping manager (scheduler)
[Nov 28, 2018 10:31:31 PM] (wangda) YARN-9061. Improve the GPU/FPGA module log 
message of
[Nov 28, 2018 10:36:30 PM] (wangda) YARN-8974. Improve the assertion message in 
TestGPUResourceHandler.
[Nov 28, 2018 10:39:06 PM] (wangda) YARN-8975. [Submarine] Use predefined 
Charset object
[Nov 28, 2018 11:03:06 PM] (wangda) YARN-8989. [YARN-8851] Move 
DockerCommandPlugin volume related APIs'
[Nov 29, 2018 1:34:14 AM] (wwei) YARN-9067. YARN Resource Manager is running 
OOM because of leak of




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.hdfs.web.TestWebHDFS 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/972/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/972/artifact/out/diff-compile-javac-root.txt
  [336K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/972/artifact/out/diff-checkstyle-root.txt
  [17M]

   hadolint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/972/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/972/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/972/artifact/out/diff-patch-pylint.txt
  [40K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/972/artifact/out/diff-patch-shellcheck.txt
  [68K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/972/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/972/artifact/out/whitespace-eol.txt
  [9.3M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/972/artifact/out/whitespace-tabs.txt
  [1.1M]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/972/artifact/out/branch-findbugs-hadoop-hdds_client.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/972/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/972/artifact/out/branch-findbugs-hadoop-hdds_framework.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/972/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/972/artifact/out/branch-findbugs-hadoop-hdds_tools.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/972/artifact/out/branch-findbugs-hadoop-ozone_client.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/972/artifact/out/branch-findbugs-hadoop-ozone_common.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/972/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt
  [8.0K]
   

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

[Takanobu Asanuma]

Thanks for driving the release, Sunil!
 
+1(non-binding)
   - verified checksums
   - succeeded in building the package (CentOS7, Java 8u181)
   - started NN HA clusters
   - verified Web UI (NN, Router)
   - verified some operations of Erasure Coding
   - verified some operations of Router-based Federation
 
Regards,
-Takanobu

on 2018/11/23 21:07, "Sunil G" wrote:

Hi folks,



Thanks to all contributors who helped in this release [1]. I have created

first release candidate (RC0) for Apache Hadoop 3.2.0.


Artifacts for this RC are available here:

http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/



RC tag in git is release-3.2.0-RC0.



The maven artifacts are available via repository.apache.org at

https://repository.apache.org/content/repositories/orgapachehadoop-1174/


This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.



3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
additions

are the highlights of this release.

1. Node Attributes Support in YARN

2. Hadoop Submarine project for running Deep Learning workloads on YARN

3. Support service upgrade via YARN Service API and CLI

4. HDFS Storage Policy Satisfier

5. Support Windows Azure Storage - Blob file system in Hadoop

6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a

7. Improvements in Router-based HDFS federation



Thanks to Wangda, Vinod, Marton for helping me in preparing the release.

I have done few testing with my pseudo cluster. My +1 to start.



Regards,

Sunil



[1]


https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E

[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
ORDER BY fixVersion ASC

[jira] [Created] (HDDS-882) Provide a config to optionally turn on/off the sync flag during chunk writes

[Shashikant Banerjee (JIRA)]

Shashikant Banerjee created HDDS-882:


 Summary: Provide a config to optionally turn on/off the sync flag 
during chunk writes
 Key: HDDS-882
 URL: https://issues.apache.org/jira/browse/HDDS-882
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
  Components: Ozone Datanode
Affects Versions: 0.4.0
Reporter: Shashikant Banerjee
Assignee: Shashikant Banerjee
 Fix For: 0.4.0


Currently, chunk writes happen with sync flag on. We should add a config to 
enable/disable this for performance benchmarks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

[Zsolt Venczel]

Thanks Sunil for preparing this release!

+1 (non-binding)

- Built from source at tag 3.2.0-rc0 (Ubuntu 16.04, JDK1.8.0_191)
- Installed on a 4-node virtual cluster
- Executed teragen/terasort/teravalidate with and without EC (using
XOR-2-1-1024k policy)
- Created two snapshots and performed snapshot diff
- Checked web UIs (NN, DN, RM)
- Run unit tests for hadoop-hdfs (found one failure case of
https://issues.apache.org/jira/browse/HDFS-12116 that is not a regression)

Best regards,
Zsolt

On Thu, Nov 29, 2018 at 6:15 AM Wilfred Spiegelenburg
 wrote:

> +1 (non binding)
>
> - build from source on MacOSX 10.14.1, 1.8.0u181
> - successful native build on Ubuntu 16.04.3
> - confirmed the checksum and signature
> - deployed a single node cluster  (jdk 1.8u191 / centos 7.5)
>
> Wilfred
>
> > On 23 Nov 2018, at 23:06, Sunil G  wrote:
> >
> > Hi folks,
> >
> >
> >
> > Thanks to all contributors who helped in this release [1]. I have created
> >
> > first release candidate (RC0) for Apache Hadoop 3.2.0.
> >
> >
> > Artifacts for this RC are available here:
> >
> > http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/
> >
> >
> >
> > RC tag in git is release-3.2.0-RC0.
> >
> >
> >
> > The maven artifacts are available via repository.apache.org at
> >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1174/
> >
> >
> > This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.
> >
> >
> >
> > 3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
> > additions
> >
> > are the highlights of this release.
> >
> > 1. Node Attributes Support in YARN
> >
> > 2. Hadoop Submarine project for running Deep Learning workloads on YARN
> >
> > 3. Support service upgrade via YARN Service API and CLI
> >
> > 4. HDFS Storage Policy Satisfier
> >
> > 5. Support Windows Azure Storage - Blob file system in Hadoop
> >
> > 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
> >
> > 7. Improvements in Router-based HDFS federation
> >
> >
> >
> > Thanks to Wangda, Vinod, Marton for helping me in preparing the release.
> >
> > I have done few testing with my pseudo cluster. My +1 to start.
> >
> >
> >
> > Regards,
> >
> > Sunil
> >
> >
> >
> > [1]
> >
> >
> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
> >
> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
> > AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
> > ORDER BY fixVersion ASC
>
>
> Wilfred Spiegelenburg | Software Engineer
> cloudera.com 
>
>
>
>
>
>
>
>