[jira] [Commented] (HDFS-12702) Ozone: Add hugo to the dev docker image
[
https://issues.apache.org/jira/browse/HDFS-12702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220058#comment-16220058
]
Elek, Marton commented on HDFS-12702:
-
I prefer to use more up-to-date version. Xenial uses 0.15 which is an old
version from ~02-2016
https://packages.ubuntu.com/search?keywords=hugo&searchon=names&suite=xenial§ion=all
The latest version is 0.30.
> Ozone: Add hugo to the dev docker image
> ---
>
> Key: HDFS-12702
> URL: https://issues.apache.org/jira/browse/HDFS-12702
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
> Attachments: HDFS-12702-HDFS-7240.001.patch,
> HDFS-12702-HDFS-7240.002.patch
>
>
> Both HADOOP-14163 and HDFS-12664 requries hugo site generation tool. To make
> it easier to review those patches I suggest to add Hugo to the dev docker
> image now.
> This patch adds hugo to the dev docker image:
> Test method:
> {code}
> cd dev-support/docker
> docker build -t test .
> docker run test hugo version
> docker rmi test
> {code}
> Expected output (after docker run):
> {code}
> Hugo Static Site Generator v0.30.2 linux/amd64 BuildDate: 2017-10-19T11:34:27Z
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12582) Replace HdfsFileStatus constructor with a builder pattern.
[ https://issues.apache.org/jira/browse/HDFS-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HDFS-12582: - Attachment: HDFS-12582.05.patch I implemented almost exactly the same pattern as part of HDFS-12681. Sorry for the overlap, I'd forgotten about this issue until your last update. Other than the names in the builder and the defaults, the builder part of both patches is nearly identical. Does this modification to your patch look OK? I'll rebase HDFS-12681 if this is OK to commit. > Replace HdfsFileStatus constructor with a builder pattern. > -- > > Key: HDFS-12582 > URL: https://issues.apache.org/jira/browse/HDFS-12582 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham > Attachments: HDFS-12582.01.patch, HDFS-12582.02.patch, > HDFS-12582.03.patch, HDFS-12582.04.patch, HDFS-12582.05.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-9806) Allow HDFS block replicas to be provided by an external storage system
[ https://issues.apache.org/jira/browse/HDFS-9806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs reassigned HDFS-9806: Assignee: (was: Ewan Higgs) > Allow HDFS block replicas to be provided by an external storage system > -- > > Key: HDFS-9806 > URL: https://issues.apache.org/jira/browse/HDFS-9806 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Chris Douglas > Attachments: HDFS-9806-design.001.pdf, HDFS-9806-design.002.pdf > > > In addition to heterogeneous media, many applications work with heterogeneous > storage systems. The guarantees and semantics provided by these systems are > often similar, but not identical to those of > [HDFS|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/index.html]. > Any client accessing multiple storage systems is responsible for reasoning > about each system independently, and must propagate/and renew credentials for > each store. > Remote stores could be mounted under HDFS. Block locations could be mapped to > immutable file regions, opaque IDs, or other tokens that represent a > consistent view of the data. While correctness for arbitrary operations > requires careful coordination between stores, in practice we can provide > workable semantics with weaker guarantees. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-9806) Allow HDFS block replicas to be provided by an external storage system
[ https://issues.apache.org/jira/browse/HDFS-9806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs reassigned HDFS-9806: Assignee: Ewan Higgs > Allow HDFS block replicas to be provided by an external storage system > -- > > Key: HDFS-9806 > URL: https://issues.apache.org/jira/browse/HDFS-9806 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Chris Douglas >Assignee: Ewan Higgs > Attachments: HDFS-9806-design.001.pdf, HDFS-9806-design.002.pdf > > > In addition to heterogeneous media, many applications work with heterogeneous > storage systems. The guarantees and semantics provided by these systems are > often similar, but not identical to those of > [HDFS|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/index.html]. > Any client accessing multiple storage systems is responsible for reasoning > about each system independently, and must propagate/and renew credentials for > each store. > Remote stores could be mounted under HDFS. Block locations could be mapped to > immutable file regions, opaque IDs, or other tokens that represent a > consistent view of the data. While correctness for arbitrary operations > requires careful coordination between stores, in practice we can provide > workable semantics with weaker guarantees. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12713) [READ] Refactor FileRegion and BlockAliasMap to separate out HDFS metadata and PROVIDED storage metadata
[ https://issues.apache.org/jira/browse/HDFS-12713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs reassigned HDFS-12713: - Assignee: Ewan Higgs > [READ] Refactor FileRegion and BlockAliasMap to separate out HDFS metadata > and PROVIDED storage metadata > > > Key: HDFS-12713 > URL: https://issues.apache.org/jira/browse/HDFS-12713 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti >Assignee: Ewan Higgs > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7240) Object store in HDFS
[ https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-7240: Attachment: HDFS-7240.003.patch > Object store in HDFS > > > Key: HDFS-7240 > URL: https://issues.apache.org/jira/browse/HDFS-7240 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Attachments: HDFS-7240.001.patch, HDFS-7240.002.patch, > HDFS-7240.003.patch, Ozone-architecture-v1.pdf, Ozonedesignupdate.pdf, > ozone_user_v0.pdf > > > This jira proposes to add object store capabilities into HDFS. > As part of the federation work (HDFS-1052) we separated block storage as a > generic storage layer. Using the Block Pool abstraction, new kinds of > namespaces can be built on top of the storage layer i.e. datanodes. > In this jira I will explore building an object store using the datanode > storage, but independent of namespace metadata. > I will soon update with a detailed design document. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12362) Ozone: write deleted block to RAFT log for consensus on datanodes
[ https://issues.apache.org/jira/browse/HDFS-12362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220162#comment-16220162 ] Weiwei Yang commented on HDFS-12362: Hi [~chenyuyun] Are you new to Ozone? If so I suggest you to pick up an easier one, this one involves block deletion code and ratis code, probably not a good candidate to start with. I did a quick look, it looks like there currently doesn't have any newbie JIRAs under HDFS-7240, I suggest you to read design doc first, get familiar with ozone internals. I will make sure when there is some newbie JIRAs created, we let them open for new contributors. Thanks > Ozone: write deleted block to RAFT log for consensus on datanodes > - > > Key: HDFS-12362 > URL: https://issues.apache.org/jira/browse/HDFS-12362 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Weiwei Yang > Labels: OzonePostMerge > > Per discussion in HDFS-12282, we need to write deleted blocks info to RAFT > log when that is ready, see more from [comment from Anu | > https://issues.apache.org/jira/browse/HDFS-12282?focusedCommentId=16136022&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16136022]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12684) Ozone: SCMMXBean NodeCount is overlapping with NodeManagerMXBean
[
https://issues.apache.org/jira/browse/HDFS-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220166#comment-16220166
]
Weiwei Yang commented on HDFS-12684:
Hi [~elek]
Thanks for the note, I might be misinterpreting this but SCM web UI does use
the patched one via {{SCMMXBean}}, the description was the output from my local
cluster before the patch. From JXM output, this is
{{StorageContainerManagerInfo}} and {{SCMNodeManagerMXBean}} produces
{{SCMNodeManagerInfo}}.
> Ozone: SCMMXBean NodeCount is overlapping with NodeManagerMXBean
>
>
> Key: HDFS-12684
> URL: https://issues.apache.org/jira/browse/HDFS-12684
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, scm
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Minor
> Fix For: HDFS-7240
>
> Attachments: HDFS-12684-HDFS-7240.001.patch
>
>
> I found this issue while reviewing HDFS-11468, from http://scm_host:9876/jmx,
> both SCM and SCMNodeManager has {{NodeCount}} metrics
> {noformat}
> {
> "name" :
> "Hadoop:service=StorageContainerManager,name=StorageContainerManagerInfo,component=ServerRuntime",
> "modelerType" : "org.apache.hadoop.ozone.scm.StorageContainerManager",
> "ClientRpcPort" : "9860",
> "DatanodeRpcPort" : "9861",
> "NodeCount" : [ {
> "key" : "STALE",
> "value" : 0
> }, {
> "key" : "DECOMMISSIONING",
> "value" : 0
> }, {
> "key" : "DECOMMISSIONED",
> "value" : 0
> }, {
> "key" : "FREE_NODE",
> "value" : 0
> }, {
> "key" : "RAFT_MEMBER",
> "value" : 0
> }, {
> "key" : "HEALTHY",
> "value" : 0
> }, {
> "key" : "DEAD",
> "value" : 0
> }, {
> "key" : "UNKNOWN",
> "value" : 0
> } ],
> "CompileInfo" : "2017-10-17T06:47Z xxx",
> "Version" : "3.1.0-SNAPSHOT, r6019a25908ce75155656f13effd8e2e53ed43461",
> "SoftwareVersion" : "3.1.0-SNAPSHOT",
> "StartedTimeInMillis" : 1508393551065
> }, {
> "name" : "Hadoop:service=SCMNodeManager,name=SCMNodeManagerInfo",
> "modelerType" : "org.apache.hadoop.ozone.scm.node.SCMNodeManager",
> "NodeCount" : [ {
> "key" : "STALE",
> "value" : 0
> }, {
> "key" : "DECOMMISSIONING",
> "value" : 0
> }, {
> "key" : "DECOMMISSIONED",
> "value" : 0
> }, {
> "key" : "FREE_NODE",
> "value" : 0
> }, {
> "key" : "RAFT_MEMBER",
> "value" : 0
> }, {
> "key" : "HEALTHY",
> "value" : 0
> }, {
> "key" : "DEAD",
> "value" : 0
> }, {
> "key" : "UNKNOWN",
> "value" : 0
> } ],
> "OutOfChillMode" : false,
> "MinimumChillModeNodes" : 1,
> "ChillModeStatus" : "Still in chill mode, waiting on nodes to report in.
> 0 nodes reported, minimal 1 nodes required."
> }
> {noformat}
> hence, propose to remove {{NodeCount}} from {{SCMMXBean}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12618) fsck -includeSnapshots reports wrong amount of total blocks
[
https://issues.apache.org/jira/browse/HDFS-12618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220202#comment-16220202
]
Wellington Chevreuil commented on HDFS-12618:
-
Thanks for the comments [~daryn].
bq. Ignoring my rejection, I'm not even sure the logic for checking just the
first block is even correct in light of truncate and append.
I was not aware of the implications of truncate/append. Given this check is
just relevant for files that had been deleted and reside on snapshots only,
would it still be a possibility for these files to be truncated/appended?
bq. I'm not a lambda expect, but creating a primitive array that requires
forced casting of indexed element appears to be an abuse of the construct. The
wrapping and unwrapping of IOEs as suppressed exceptions embedded in
RuntimeExceptions in an apparent attempt to thwart checked dependencies is also
unnecessary and appears related to the lamba construct.
That was an attempt to use built-in *java.util.function.Consumer* interface,
that defines only one parameter on its accept method and throws no Exception.
Indeed, looking further into lambda API, I guess this can be sorted by creating
an own *@FunctionalInterface* that defines the types and checked exceptions
needed by *check* and *checkFilesInSnapshotOnly* methods.
> fsck -includeSnapshots reports wrong amount of total blocks
> ---
>
> Key: HDFS-12618
> URL: https://issues.apache.org/jira/browse/HDFS-12618
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Affects Versions: 3.0.0-alpha3
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-121618.initial, HDFS-12618.001.patch,
> HDFS-12618.002.patch
>
>
> When snapshot is enabled, if a file is deleted but is contained by a
> snapshot, *fsck* will not reported blocks for such file, showing different
> number of *total blocks* than what is exposed in the Web UI.
> This should be fine, as *fsck* provides *-includeSnapshots* option. The
> problem is that *-includeSnapshots* option causes *fsck* to count blocks for
> every occurrence of a file on snapshots, which is wrong because these blocks
> should be counted only once (for instance, if a 100MB file is present on 3
> snapshots, it would still map to one block only in hdfs). This causes fsck to
> report much more blocks than what actually exist in hdfs and is reported in
> the Web UI.
> Here's an example:
> 1) HDFS has two files of 2 blocks each:
> {noformat}
> $ hdfs dfs -ls -R /
> drwxr-xr-x - root supergroup 0 2017-10-07 21:21 /snap-test
> -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:16 /snap-test/file1
> -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:17 /snap-test/file2
> drwxr-xr-x - root supergroup 0 2017-05-13 13:03 /test
> {noformat}
> 2) There are two snapshots, with the two files present on each of the
> snapshots:
> {noformat}
> $ hdfs dfs -ls -R /snap-test/.snapshot
> drwxr-xr-x - root supergroup 0 2017-10-07 21:21
> /snap-test/.snapshot/snap1
> -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:16
> /snap-test/.snapshot/snap1/file1
> -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:17
> /snap-test/.snapshot/snap1/file2
> drwxr-xr-x - root supergroup 0 2017-10-07 21:21
> /snap-test/.snapshot/snap2
> -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:16
> /snap-test/.snapshot/snap2/file1
> -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:17
> /snap-test/.snapshot/snap2/file2
> {noformat}
> 3) *fsck -includeSnapshots* reports 12 blocks in total (4 blocks for the
> normal file path, plus 4 blocks for each snapshot path):
> {noformat}
> $ hdfs fsck / -includeSnapshots
> FSCK started by root (auth:SIMPLE) from /127.0.0.1 for path / at Mon Oct 09
> 15:15:36 BST 2017
> Status: HEALTHY
> Number of data-nodes:1
> Number of racks: 1
> Total dirs: 6
> Total symlinks: 0
> Replicated Blocks:
> Total size: 1258291200 B
> Total files: 6
> Total blocks (validated):12 (avg. block size 104857600 B)
> Minimally replicated blocks: 12 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 0 (0.0 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 1
> Average block replication: 1.0
> Missing blocks: 0
> Corrupt blocks: 0
> Missing replicas:0 (0.0 %)
> {noformat}
> 4) Web UI shows the correct number (4 blocks only):
> {noformat}
> Security is off.
> Safemode is off.
> 5 files and directories, 4 blocks = 9 total filesystem object(s).
> {noformat}
> I would like to work on this solution, will propose an ini
[jira] [Commented] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220215#comment-16220215
]
Ewan Higgs commented on HDFS-11902:
---
+1. The new patch looks good. The only difference is that code is moved as
[~virajith] said.
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11754) Make FsServerDefaults cache configurable.
[
https://issues.apache.org/jira/browse/HDFS-11754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220218#comment-16220218
]
Hadoop QA commented on HDFS-11754:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 11s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
30s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}118m 9s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}182m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestPread |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.federation.metrics.TestFederationMetrics |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-11754 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12893905/HDFS-11754.006.patch |
| Optional Tests | asflic
[jira] [Updated] (HDFS-10285) Storage Policy Satisfier in Namenode
[ https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R updated HDFS-10285: Attachment: Storage-Policy-Satisfier-in-HDFS-Oct-26-2017.pdf Uploaded another version of SPS design document, tried capturing the details to reflect recent code changes. Welcome feedback! Thank you [~umamaheswararao] for co-authoring the doc. Thank you [~andrew.wang], [~surendrasingh], [~eddyxu], [~xiaochen], [~anoop.hbase] for the useful discussions/comments. > Storage Policy Satisfier in Namenode > > > Key: HDFS-10285 > URL: https://issues.apache.org/jira/browse/HDFS-10285 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10285-consolidated-merge-patch-00.patch, > HDFS-10285-consolidated-merge-patch-01.patch, > HDFS-SPS-TestReport-20170708.pdf, > Storage-Policy-Satisfier-in-HDFS-June-20-2017.pdf, > Storage-Policy-Satisfier-in-HDFS-May10.pdf, > Storage-Policy-Satisfier-in-HDFS-Oct-26-2017.pdf > > > Heterogeneous storage in HDFS introduced the concept of storage policy. These > policies can be set on directory/file to specify the user preference, where > to store the physical block. When user set the storage policy before writing > data, then the blocks could take advantage of storage policy preferences and > stores physical block accordingly. > If user set the storage policy after writing and completing the file, then > the blocks would have been written with default storage policy (nothing but > DISK). User has to run the ‘Mover tool’ explicitly by specifying all such > file names as a list. In some distributed system scenarios (ex: HBase) it > would be difficult to collect all the files and run the tool as different > nodes can write files separately and file can have different paths. > Another scenarios is, when user rename the files from one effected storage > policy file (inherited policy from parent directory) to another storage > policy effected directory, it will not copy inherited storage policy from > source. So it will take effect from destination file/dir parent storage > policy. This rename operation is just a metadata change in Namenode. The > physical blocks still remain with source storage policy. > So, Tracking all such business logic based file names could be difficult for > admins from distributed nodes(ex: region servers) and running the Mover tool. > Here the proposal is to provide an API from Namenode itself for trigger the > storage policy satisfaction. A Daemon thread inside Namenode should track > such calls and process to DN as movement commands. > Will post the detailed design thoughts document soon. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10285) Storage Policy Satisfier in Namenode
[ https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220276#comment-16220276 ] Rakesh R edited comment on HDFS-10285 at 10/26/17 10:35 AM: Uploaded another version of SPS design document, tried capturing the details to reflect recent code changes. Welcome feedback! Thank you [~umamaheswararao] for co-authoring the doc. Thank you [~andrew.wang], [~surendrasingh], [~eddyxu], [~xiaochen], [~anoop.hbase], [~ram_krish]] for the useful discussions/comments. was (Author: rakeshr): Uploaded another version of SPS design document, tried capturing the details to reflect recent code changes. Welcome feedback! Thank you [~umamaheswararao] for co-authoring the doc. Thank you [~andrew.wang], [~surendrasingh], [~eddyxu], [~xiaochen], [~anoop.hbase] for the useful discussions/comments. > Storage Policy Satisfier in Namenode > > > Key: HDFS-10285 > URL: https://issues.apache.org/jira/browse/HDFS-10285 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10285-consolidated-merge-patch-00.patch, > HDFS-10285-consolidated-merge-patch-01.patch, > HDFS-SPS-TestReport-20170708.pdf, > Storage-Policy-Satisfier-in-HDFS-June-20-2017.pdf, > Storage-Policy-Satisfier-in-HDFS-May10.pdf, > Storage-Policy-Satisfier-in-HDFS-Oct-26-2017.pdf > > > Heterogeneous storage in HDFS introduced the concept of storage policy. These > policies can be set on directory/file to specify the user preference, where > to store the physical block. When user set the storage policy before writing > data, then the blocks could take advantage of storage policy preferences and > stores physical block accordingly. > If user set the storage policy after writing and completing the file, then > the blocks would have been written with default storage policy (nothing but > DISK). User has to run the ‘Mover tool’ explicitly by specifying all such > file names as a list. In some distributed system scenarios (ex: HBase) it > would be difficult to collect all the files and run the tool as different > nodes can write files separately and file can have different paths. > Another scenarios is, when user rename the files from one effected storage > policy file (inherited policy from parent directory) to another storage > policy effected directory, it will not copy inherited storage policy from > source. So it will take effect from destination file/dir parent storage > policy. This rename operation is just a metadata change in Namenode. The > physical blocks still remain with source storage policy. > So, Tracking all such business logic based file names could be difficult for > admins from distributed nodes(ex: region servers) and running the Mover tool. > Here the proposal is to provide an API from Namenode itself for trigger the > storage policy satisfaction. A Daemon thread inside Namenode should track > such calls and process to DN as movement commands. > Will post the detailed design thoughts document soon. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12715) I didn't find the "-find" command on hadoop2.6
Djc created HDFS-12715: -- Summary: I didn't find the "-find" command on hadoop2.6 Key: HDFS-12715 URL: https://issues.apache.org/jira/browse/HDFS-12715 Project: Hadoop HDFS Issue Type: Bug Components: auto-failover Affects Versions: 2.6.0 Reporter: Djc Priority: Critical Fix For: 2.6.0 When I looked for files on HDFS, I found no "find" command. I didn't find the "find" command by using "Hadoop FS -help". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12716) 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes to be available
usharani created HDFS-12716: --- Summary: 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes to be available Key: HDFS-12716 URL: https://issues.apache.org/jira/browse/HDFS-12716 Project: Hadoop HDFS Issue Type: Bug Components: datanode Reporter: usharani Currently 'dfs.datanode.failed.volumes.tolerated' supports number of tolerated failed volumes to be mentioned. This configuration change requires restart of datanode. Since datanode volumes can be changed dynamically, keeping this configuration same for all may not be good idea. Support 'dfs.datanode.failed.volumes.tolerated' to accept special 'negative value 'x' to tolerate failures of upto "n-x" -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12716) 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes to be available
[ https://issues.apache.org/jira/browse/HDFS-12716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] usharani reassigned HDFS-12716: --- Assignee: usharani > 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes > to be available > - > > Key: HDFS-12716 > URL: https://issues.apache.org/jira/browse/HDFS-12716 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: usharani >Assignee: usharani > > Currently 'dfs.datanode.failed.volumes.tolerated' supports number of > tolerated failed volumes to be mentioned. This configuration change requires > restart of datanode. Since datanode volumes can be changed dynamically, > keeping this configuration same for all may not be good idea. > Support 'dfs.datanode.failed.volumes.tolerated' to accept special > 'negative value 'x' to tolerate failures of upto "n-x" -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-12715) I didn't find the "-find" command on hadoop2.6
[ https://issues.apache.org/jira/browse/HDFS-12715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HDFS-12715. Resolution: Invalid Fix Version/s: (was: 2.6.0) Please use the user@hadoop mailing list for these kinds of questions: https://lists.apache.org/list.html?u...@hadoop.apache.org > I didn't find the "-find" command on hadoop2.6 > -- > > Key: HDFS-12715 > URL: https://issues.apache.org/jira/browse/HDFS-12715 > Project: Hadoop HDFS > Issue Type: Bug > Components: auto-failover >Affects Versions: 2.6.0 >Reporter: Djc >Priority: Critical > > When I looked for files on HDFS, I found no "find" command. I didn't find the > "find" command by using "Hadoop FS -help". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12443) Ozone: Improve SCM block deletion throttling algorithm
[
https://issues.apache.org/jira/browse/HDFS-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-12443:
-
Attachment: HDFS-12443-HDFS-7240.001.patch
> Ozone: Improve SCM block deletion throttling algorithm
> ---
>
> Key: HDFS-12443
> URL: https://issues.apache.org/jira/browse/HDFS-12443
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, scm
>Reporter: Weiwei Yang
>Assignee: Yiqun Lin
> Labels: OzonePostMerge
> Attachments: HDFS-12443-HDFS-7240.001.patch
>
>
> Currently SCM scans delLog to send deletion transactions to datanode
> periodically, the throttling algorithm is simple, it scans at most
> {{BLOCK_DELETE_TX_PER_REQUEST_LIMIT}} (by default 50) at a time. This is
> non-optimal, worst case it might cache 50 TXs for 50 different DNs so each DN
> will only get 1 TX to proceed in an interval, this will make the deletion
> slow. An improvement to this is to make this throttling by datanode, e.g 50
> TXs per datanode per interval.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12443) Ozone: Improve SCM block deletion throttling algorithm
[
https://issues.apache.org/jira/browse/HDFS-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220394#comment-16220394
]
Yiqun Lin commented on HDFS-12443:
--
Hi [~cheersyang],
bq. But this approach makes it a bit difficult to prepare a reasonable size of
TXs for DNs, please take a look at this issue and let me know if you have any
idea to resolve it.
I think this should the same type problem that we discussed in HDFS-12691.
Maybe as you said, we should do some scaling testing, then we get a appropriate
value.
Let's get back of this JIRA, today I took some time making some change of block
deletion throttling algorithm.
Attach the initial patch (With some additional log floodings fixed).
[~cheersyang], please have a review.
> Ozone: Improve SCM block deletion throttling algorithm
> ---
>
> Key: HDFS-12443
> URL: https://issues.apache.org/jira/browse/HDFS-12443
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, scm
>Reporter: Weiwei Yang
>Assignee: Yiqun Lin
> Labels: OzonePostMerge
> Attachments: HDFS-12443-HDFS-7240.001.patch
>
>
> Currently SCM scans delLog to send deletion transactions to datanode
> periodically, the throttling algorithm is simple, it scans at most
> {{BLOCK_DELETE_TX_PER_REQUEST_LIMIT}} (by default 50) at a time. This is
> non-optimal, worst case it might cache 50 TXs for 50 different DNs so each DN
> will only get 1 TX to proceed in an interval, this will make the deletion
> slow. An improvement to this is to make this throttling by datanode, e.g 50
> TXs per datanode per interval.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12443) Ozone: Improve SCM block deletion throttling algorithm
[
https://issues.apache.org/jira/browse/HDFS-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-12443:
-
Status: Patch Available (was: Open)
> Ozone: Improve SCM block deletion throttling algorithm
> ---
>
> Key: HDFS-12443
> URL: https://issues.apache.org/jira/browse/HDFS-12443
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, scm
>Reporter: Weiwei Yang
>Assignee: Yiqun Lin
> Labels: OzonePostMerge
> Attachments: HDFS-12443-HDFS-7240.001.patch
>
>
> Currently SCM scans delLog to send deletion transactions to datanode
> periodically, the throttling algorithm is simple, it scans at most
> {{BLOCK_DELETE_TX_PER_REQUEST_LIMIT}} (by default 50) at a time. This is
> non-optimal, worst case it might cache 50 TXs for 50 different DNs so each DN
> will only get 1 TX to proceed in an interval, this will make the deletion
> slow. An improvement to this is to make this throttling by datanode, e.g 50
> TXs per datanode per interval.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12717) 'hdfs namenode -help' command will not work when Namenode is Running
Harshakiran Reddy created HDFS-12717: Summary: 'hdfs namenode -help' command will not work when Namenode is Running Key: HDFS-12717 URL: https://issues.apache.org/jira/browse/HDFS-12717 Project: Hadoop HDFS Issue Type: Bug Components: scripts Affects Versions: 3.0.0-alpha1 Reporter: Harshakiran Reddy Scenario: Start namenode Execute "hdfs namenode -help" Actul Output: = namenode is running as process 85785. Stop it first. Expected output: === Should Give the help message -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12697) Ozone services must stay disabled in secure setup for alpha
[
https://issues.apache.org/jira/browse/HDFS-12697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220475#comment-16220475
]
Elek, Marton commented on HDFS-12697:
-
I am not sure it works well.
It will return a security error even if ozone is turned off and not the
security on.
With ozone.enabled !=false, but security is turned off:
{code}
scm_1 | 2017-10-26 14:03:17 ERROR StorageContainerManager:320 - SCM
cannot be started in secure mode
scm_1 | SCM cannot be started in secure mode
{code}
> Ozone services must stay disabled in secure setup for alpha
> ---
>
> Key: HDFS-12697
> URL: https://issues.apache.org/jira/browse/HDFS-12697
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jitendra Nath Pandey
>Assignee: Bharat Viswanadham
>Priority: Blocker
> Attachments: HDFS-12697-HDFS-7240.01.patch,
> HDFS-12697-HDFS-7240.02.patch
>
>
> When security is enabled, ozone services should not start up, even if ozone
> configurations are enabled. This is important to ensure a user experimenting
> with ozone doesn't inadvertently get exposed to attacks. Specifically,
> 1) KSM should not start up.
> 2) SCM should not startup.
> 3) Datanode's ozone xceiverserver should not startup, and must not listen on
> a port.
> 4) Datanode's ozone handler port should not be open, and webservice must stay
> disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12521) Ozone: SCM should read all Container info into memory when booting up
[ https://issues.apache.org/jira/browse/HDFS-12521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDFS-12521: --- Attachment: HDFS-12521-HDFS-7240.007.patch Uploaded v7 patch again for jenkins re-trigger. > Ozone: SCM should read all Container info into memory when booting up > - > > Key: HDFS-12521 > URL: https://issues.apache.org/jira/browse/HDFS-12521 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Lokesh Jain > Labels: ozoneMerge > Attachments: HDFS-12521-HDFS-7240.001.patch, > HDFS-12521-HDFS-7240.002.patch, HDFS-12521-HDFS-7240.003.patch, > HDFS-12521-HDFS-7240.004.patch, HDFS-12521-HDFS-7240.005.patch, > HDFS-12521-HDFS-7240.006.patch, HDFS-12521-HDFS-7240.007.patch, > HDFS-12521-HDFS-7240.007.patch > > > When SCM boots up it should read all containers into memory. This is a > performance optimization that allows delays on SCM side. This JIRA tracks > that issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12700) Fix datanode link that can not be accessed in dfshealth.html
[ https://issues.apache.org/jira/browse/HDFS-12700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220525#comment-16220525 ] fang zhenyi commented on HDFS-12700: [~arpitagarwal]thanks. > Fix datanode link that can not be accessed in dfshealth.html > -- > > Key: HDFS-12700 > URL: https://issues.apache.org/jira/browse/HDFS-12700 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: fang zhenyi >Assignee: fang zhenyi >Priority: Minor > Fix For: 3.1.0 > > Attachments: HDFS-12700.000.patch > > > I find that datanode link that can not be accessed in dfshealth.html if I > do not change hosts file.So I changed the link to ip address. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12697) Ozone services must stay disabled in secure setup for alpha
[
https://issues.apache.org/jira/browse/HDFS-12697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220475#comment-16220475
]
Elek, Marton edited comment on HDFS-12697 at 10/26/17 2:48 PM:
---
I am not sure it works well.
It will return a security error even if ozone is turned off and not the
security on.
With ozone.enabled !=false, but security is turned off:
{code}
scm_1 | 2017-10-26 14:03:17 ERROR StorageContainerManager:320 - SCM
cannot be started in secure mode
scm_1 | SCM cannot be started in secure mode
{code}
UPDATE: I found the reviewboard and Xiaoyu wrote the same...
was (Author: elek):
I am not sure it works well.
It will return a security error even if ozone is turned off and not the
security on.
With ozone.enabled !=false, but security is turned off:
{code}
scm_1 | 2017-10-26 14:03:17 ERROR StorageContainerManager:320 - SCM
cannot be started in secure mode
scm_1 | SCM cannot be started in secure mode
{code}
> Ozone services must stay disabled in secure setup for alpha
> ---
>
> Key: HDFS-12697
> URL: https://issues.apache.org/jira/browse/HDFS-12697
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jitendra Nath Pandey
>Assignee: Bharat Viswanadham
>Priority: Blocker
> Attachments: HDFS-12697-HDFS-7240.01.patch,
> HDFS-12697-HDFS-7240.02.patch
>
>
> When security is enabled, ozone services should not start up, even if ozone
> configurations are enabled. This is important to ensure a user experimenting
> with ozone doesn't inadvertently get exposed to attacks. Specifically,
> 1) KSM should not start up.
> 2) SCM should not startup.
> 3) Datanode's ozone xceiverserver should not startup, and must not listen on
> a port.
> 4) Datanode's ozone handler port should not be open, and webservice must stay
> disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12698) Ozone: Use time units in the Ozone configuration values
[
https://issues.apache.org/jira/browse/HDFS-12698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDFS-12698:
Status: Patch Available (was: Open)
> Ozone: Use time units in the Ozone configuration values
> ---
>
> Key: HDFS-12698
> URL: https://issues.apache.org/jira/browse/HDFS-12698
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
> Attachments: HDFS-12698-HDFS-7240.001.patch,
> HDFS-12698-HDFS-7240.002.patch
>
>
> In HDFS-9847 introduced a new way to configure the time related configuration
> with using time unit in the vaule (eg. 10s, 5m, ...).
> Because the new behavior I have seen a lot of warning during my tests:
> {code}
> 2017-10-19 18:35:19,955 [main] INFO Configuration.deprecation
> (Configuration.java:logDeprecation(1306)) - No unit for
> scm.container.client.idle.threshold(1) assuming MILLISECONDS
> {code}
> So we need to add the time unit for every configuration. Unfortunately we
> have a few configuration parameter which includes the unit in the key name
> (eg dfs.cblock.block.buffer.flush.interval.seconds or
> ozone.container.report.interval.ms).
> I suggest to remove all the units from the key name and follow the new
> convention where any of the units could be used.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12698) Ozone: Use time units in the Ozone configuration values
[
https://issues.apache.org/jira/browse/HDFS-12698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDFS-12698:
Attachment: HDFS-12698-HDFS-7240.002.patch
> Ozone: Use time units in the Ozone configuration values
> ---
>
> Key: HDFS-12698
> URL: https://issues.apache.org/jira/browse/HDFS-12698
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
> Attachments: HDFS-12698-HDFS-7240.001.patch,
> HDFS-12698-HDFS-7240.002.patch
>
>
> In HDFS-9847 introduced a new way to configure the time related configuration
> with using time unit in the vaule (eg. 10s, 5m, ...).
> Because the new behavior I have seen a lot of warning during my tests:
> {code}
> 2017-10-19 18:35:19,955 [main] INFO Configuration.deprecation
> (Configuration.java:logDeprecation(1306)) - No unit for
> scm.container.client.idle.threshold(1) assuming MILLISECONDS
> {code}
> So we need to add the time unit for every configuration. Unfortunately we
> have a few configuration parameter which includes the unit in the key name
> (eg dfs.cblock.block.buffer.flush.interval.seconds or
> ozone.container.report.interval.ms).
> I suggest to remove all the units from the key name and follow the new
> convention where any of the units could be used.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12718) Ozone: fix thread number calculation in CBlockManager
[
https://issues.apache.org/jira/browse/HDFS-12718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton reassigned HDFS-12718:
---
Assignee: Elek, Marton
> Ozone: fix thread number calculation in CBlockManager
> -
>
> Key: HDFS-12718
> URL: https://issues.apache.org/jira/browse/HDFS-12718
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>
> When starting cblock server or during the unit tests I got many
> IllegalArgumentException:
> {code}
> testCliInfoVolume(org.apache.hadoop.cblock.TestCBlockCLI) Time elapsed:
> 0.004 sec <<< ERROR!
> org.apache.hadoop.ipc.RemoteException: java.lang.IllegalArgumentException
> at
> java.util.concurrent.ThreadPoolExecutor.(ThreadPoolExecutor.java:1307)
> at
> java.util.concurrent.ThreadPoolExecutor.(ThreadPoolExecutor.java:1265)
> at
> org.apache.hadoop.cblock.storage.StorageManager.createVolumeContainers(StorageManager.java:212)
> at
> org.apache.hadoop.cblock.storage.StorageManager.createVolume(StorageManager.java:304)
> at
> org.apache.hadoop.cblock.CBlockManager.createVolume(CBlockManager.java:257)
> at
> org.apache.hadoop.cblock.protocolPB.CBlockServiceProtocolServerSideTranslatorPB.createVolume(CBlockServiceProtocolServerSideTranslatorPB.java:57)
> at
> org.apache.hadoop.cblock.protocol.proto.CBlockServiceProtocolProtos$CBlockServiceProtocolService$2.callBlockingMethod(CBlockServiceProtocolProtos.java:6056)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491)
> at org.apache.hadoop.ipc.Client.call(Client.java:1437)
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy10.createVolume(Unknown Source)
> at
> org.apache.hadoop.cblock.client.CBlockServiceProtocolClientSideTranslatorPB.createVolume(CBlockServiceProtocolClientSideTranslatorPB.java:64)
> at
> org.apache.hadoop.cblock.client.CBlockVolumeClient.createVolume(CBlockVolumeClient.java:64)
> at
> org.apache.hadoop.cblock.cli.CBlockCli.createVolume(CBlockCli.java:239)
> at org.apache.hadoop.cblock.cli.CBlockCli.run(CBlockCli.java:173)
> at
> org.apache.hadoop.cblock.TestCBlockCLI.testCliInfoVolume(TestCBlockCLI.java:232)
> {code}
> The root cause is that in CBlock Manager we create ThreadGroups:
> {code}
> ThreadPoolExecutor executor = new ThreadPoolExecutor(numThreads,
> MAX_THREADS, 1, TimeUnit.SECONDS,
> new ArrayBlockingQueue<>(MAX_QUEUE_CAPACITY),
> new ThreadPoolExecutor.CallerRunsPolicy());
> {code}
> Where numThreads (the number of always active threads) comes from config and
> 16 by default MAX_THREADS is `Runtime.getRuntime().availableProcessors() * 2`.
> My problem was that MAX_THREAD was lower than numThreads (as I have only 2
> processors, shame on me), so I got IllegalArgumentException.
> In the fix I suggest:
> * Limit the maximum number of threads not the always active threads, as
> ususally this is the number which is needed to adjust
> * Use the core dependent numbers as the number of active threas (but if
> numThreads is smaller, that should be used).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12718) Ozone: fix thread number calculation in CBlockManager
Elek, Marton created HDFS-12718:
---
Summary: Ozone: fix thread number calculation in CBlockManager
Key: HDFS-12718
URL: https://issues.apache.org/jira/browse/HDFS-12718
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Affects Versions: HDFS-7240
Reporter: Elek, Marton
When starting cblock server or during the unit tests I got many
IllegalArgumentException:
{code}
testCliInfoVolume(org.apache.hadoop.cblock.TestCBlockCLI) Time elapsed: 0.004
sec <<< ERROR!
org.apache.hadoop.ipc.RemoteException: java.lang.IllegalArgumentException
at
java.util.concurrent.ThreadPoolExecutor.(ThreadPoolExecutor.java:1307)
at
java.util.concurrent.ThreadPoolExecutor.(ThreadPoolExecutor.java:1265)
at
org.apache.hadoop.cblock.storage.StorageManager.createVolumeContainers(StorageManager.java:212)
at
org.apache.hadoop.cblock.storage.StorageManager.createVolume(StorageManager.java:304)
at
org.apache.hadoop.cblock.CBlockManager.createVolume(CBlockManager.java:257)
at
org.apache.hadoop.cblock.protocolPB.CBlockServiceProtocolServerSideTranslatorPB.createVolume(CBlockServiceProtocolServerSideTranslatorPB.java:57)
at
org.apache.hadoop.cblock.protocol.proto.CBlockServiceProtocolProtos$CBlockServiceProtocolService$2.callBlockingMethod(CBlockServiceProtocolProtos.java:6056)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491)
at org.apache.hadoop.ipc.Client.call(Client.java:1437)
at org.apache.hadoop.ipc.Client.call(Client.java:1347)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy10.createVolume(Unknown Source)
at
org.apache.hadoop.cblock.client.CBlockServiceProtocolClientSideTranslatorPB.createVolume(CBlockServiceProtocolClientSideTranslatorPB.java:64)
at
org.apache.hadoop.cblock.client.CBlockVolumeClient.createVolume(CBlockVolumeClient.java:64)
at
org.apache.hadoop.cblock.cli.CBlockCli.createVolume(CBlockCli.java:239)
at org.apache.hadoop.cblock.cli.CBlockCli.run(CBlockCli.java:173)
at
org.apache.hadoop.cblock.TestCBlockCLI.testCliInfoVolume(TestCBlockCLI.java:232)
{code}
The root cause is that in CBlock Manager we create ThreadGroups:
{code}
ThreadPoolExecutor executor = new ThreadPoolExecutor(numThreads,
MAX_THREADS, 1, TimeUnit.SECONDS,
new ArrayBlockingQueue<>(MAX_QUEUE_CAPACITY),
new ThreadPoolExecutor.CallerRunsPolicy());
{code}
Where numThreads (the number of always active threads) comes from config and 16
by default MAX_THREADS is `Runtime.getRuntime().availableProcessors() * 2`.
My problem was that MAX_THREAD was lower than numThreads (as I have only 2
processors, shame on me), so I got IllegalArgumentException.
In the fix I suggest:
* Limit the maximum number of threads not the always active threads, as
ususally this is the number which is needed to adjust
* Use the core dependent numbers as the number of active threas (but if
numThreads is smaller, that should be used).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12718) Ozone: fix thread number calculation in CBlockManager
[
https://issues.apache.org/jira/browse/HDFS-12718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDFS-12718:
Attachment: HDFS-12718-HDFS-7240.001.patch
> Ozone: fix thread number calculation in CBlockManager
> -
>
> Key: HDFS-12718
> URL: https://issues.apache.org/jira/browse/HDFS-12718
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
> Attachments: HDFS-12718-HDFS-7240.001.patch
>
>
> When starting cblock server or during the unit tests I got many
> IllegalArgumentException:
> {code}
> testCliInfoVolume(org.apache.hadoop.cblock.TestCBlockCLI) Time elapsed:
> 0.004 sec <<< ERROR!
> org.apache.hadoop.ipc.RemoteException: java.lang.IllegalArgumentException
> at
> java.util.concurrent.ThreadPoolExecutor.(ThreadPoolExecutor.java:1307)
> at
> java.util.concurrent.ThreadPoolExecutor.(ThreadPoolExecutor.java:1265)
> at
> org.apache.hadoop.cblock.storage.StorageManager.createVolumeContainers(StorageManager.java:212)
> at
> org.apache.hadoop.cblock.storage.StorageManager.createVolume(StorageManager.java:304)
> at
> org.apache.hadoop.cblock.CBlockManager.createVolume(CBlockManager.java:257)
> at
> org.apache.hadoop.cblock.protocolPB.CBlockServiceProtocolServerSideTranslatorPB.createVolume(CBlockServiceProtocolServerSideTranslatorPB.java:57)
> at
> org.apache.hadoop.cblock.protocol.proto.CBlockServiceProtocolProtos$CBlockServiceProtocolService$2.callBlockingMethod(CBlockServiceProtocolProtos.java:6056)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491)
> at org.apache.hadoop.ipc.Client.call(Client.java:1437)
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy10.createVolume(Unknown Source)
> at
> org.apache.hadoop.cblock.client.CBlockServiceProtocolClientSideTranslatorPB.createVolume(CBlockServiceProtocolClientSideTranslatorPB.java:64)
> at
> org.apache.hadoop.cblock.client.CBlockVolumeClient.createVolume(CBlockVolumeClient.java:64)
> at
> org.apache.hadoop.cblock.cli.CBlockCli.createVolume(CBlockCli.java:239)
> at org.apache.hadoop.cblock.cli.CBlockCli.run(CBlockCli.java:173)
> at
> org.apache.hadoop.cblock.TestCBlockCLI.testCliInfoVolume(TestCBlockCLI.java:232)
> {code}
> The root cause is that in CBlock Manager we create ThreadGroups:
> {code}
> ThreadPoolExecutor executor = new ThreadPoolExecutor(numThreads,
> MAX_THREADS, 1, TimeUnit.SECONDS,
> new ArrayBlockingQueue<>(MAX_QUEUE_CAPACITY),
> new ThreadPoolExecutor.CallerRunsPolicy());
> {code}
> Where numThreads (the number of always active threads) comes from config and
> 16 by default MAX_THREADS is `Runtime.getRuntime().availableProcessors() * 2`.
> My problem was that MAX_THREAD was lower than numThreads (as I have only 2
> processors, shame on me), so I got IllegalArgumentException.
> In the fix I suggest:
> * Limit the maximum number of threads not the always active threads, as
> ususally this is the number which is needed to adjust
> * Use the core dependent numbers as the number of active threas (but if
> numThreads is smaller, that should be used).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12718) Ozone: fix thread number calculation in CBlockManager
[
https://issues.apache.org/jira/browse/HDFS-12718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDFS-12718:
Status: Patch Available (was: Open)
> Ozone: fix thread number calculation in CBlockManager
> -
>
> Key: HDFS-12718
> URL: https://issues.apache.org/jira/browse/HDFS-12718
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
> Attachments: HDFS-12718-HDFS-7240.001.patch
>
>
> When starting cblock server or during the unit tests I got many
> IllegalArgumentException:
> {code}
> testCliInfoVolume(org.apache.hadoop.cblock.TestCBlockCLI) Time elapsed:
> 0.004 sec <<< ERROR!
> org.apache.hadoop.ipc.RemoteException: java.lang.IllegalArgumentException
> at
> java.util.concurrent.ThreadPoolExecutor.(ThreadPoolExecutor.java:1307)
> at
> java.util.concurrent.ThreadPoolExecutor.(ThreadPoolExecutor.java:1265)
> at
> org.apache.hadoop.cblock.storage.StorageManager.createVolumeContainers(StorageManager.java:212)
> at
> org.apache.hadoop.cblock.storage.StorageManager.createVolume(StorageManager.java:304)
> at
> org.apache.hadoop.cblock.CBlockManager.createVolume(CBlockManager.java:257)
> at
> org.apache.hadoop.cblock.protocolPB.CBlockServiceProtocolServerSideTranslatorPB.createVolume(CBlockServiceProtocolServerSideTranslatorPB.java:57)
> at
> org.apache.hadoop.cblock.protocol.proto.CBlockServiceProtocolProtos$CBlockServiceProtocolService$2.callBlockingMethod(CBlockServiceProtocolProtos.java:6056)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491)
> at org.apache.hadoop.ipc.Client.call(Client.java:1437)
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy10.createVolume(Unknown Source)
> at
> org.apache.hadoop.cblock.client.CBlockServiceProtocolClientSideTranslatorPB.createVolume(CBlockServiceProtocolClientSideTranslatorPB.java:64)
> at
> org.apache.hadoop.cblock.client.CBlockVolumeClient.createVolume(CBlockVolumeClient.java:64)
> at
> org.apache.hadoop.cblock.cli.CBlockCli.createVolume(CBlockCli.java:239)
> at org.apache.hadoop.cblock.cli.CBlockCli.run(CBlockCli.java:173)
> at
> org.apache.hadoop.cblock.TestCBlockCLI.testCliInfoVolume(TestCBlockCLI.java:232)
> {code}
> The root cause is that in CBlock Manager we create ThreadGroups:
> {code}
> ThreadPoolExecutor executor = new ThreadPoolExecutor(numThreads,
> MAX_THREADS, 1, TimeUnit.SECONDS,
> new ArrayBlockingQueue<>(MAX_QUEUE_CAPACITY),
> new ThreadPoolExecutor.CallerRunsPolicy());
> {code}
> Where numThreads (the number of always active threads) comes from config and
> 16 by default MAX_THREADS is `Runtime.getRuntime().availableProcessors() * 2`.
> My problem was that MAX_THREAD was lower than numThreads (as I have only 2
> processors, shame on me), so I got IllegalArgumentException.
> In the fix I suggest:
> * Limit the maximum number of threads not the always active threads, as
> ususally this is the number which is needed to adjust
> * Use the core dependent numbers as the number of active threas (but if
> numThreads is smaller, that should be used).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12618) fsck -includeSnapshots reports wrong amount of total blocks
[
https://issues.apache.org/jira/browse/HDFS-12618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220685#comment-16220685
]
Daryn Sharp commented on HDFS-12618:
bq. I did not -1 on this one because this only happens when someone runs
-includeSnapshots explicitly. Not sure if these snapshot problems can be solved
without doing this, please feel free to share any alternatives in mind.
Been awhile since I (attempted to) analyze snapshots. I think snapshotted
files, even deleted ones, are a {{INodeReference.WithName}} with a parent of
{{INodeReference.WithCount}} which maintains a list of all
{{INodeReference.WithName}}. Perhaps we could detect whether the inode is
linked into the current namesystem. If yes, it will be picked up in the
namesystem crawl; if no, count it based on all the {{WithName}} refs. And/or
maybe count a reference only if it's the last ref
({{INodeReference.WithCount#getLastWithName}}). Maybe.
bq. For large clusters doing fsck alone on / are a bad idea.
We do this nightly. Every day. Every cluster.
bq. Would it work for you if we put a memory limit on how much each fsck
-includeSnapshots' block map could consume on the NN?
I'm not sure how that could work in a user-friendly manner. I run the fsck, it
fails. I have to run it again on subdirs? Some fail again. I have to run it
on lower subdirs, then write code to collate all the mini-reports back together
in a unified report?
Fsck can run for tens of minutes or hours. Keeping excessively large state
during the operation will cause lots of pressure on old and risk OOM. It has
to stay lightweight (or only as heavy as it already is).
bq. Given this check is just relevant for files that had been deleted and
reside on snapshots only, would it still be a possibility for these files to be
truncated/appended?
I'm by no means a truncate/append + snapshot expert. A renamed file appears as
a delete in a snapshot diff. It can be subsequent modified in later versions
that may also be snapshotted and "deleted". Correct snapshot handling has been
problematic so we need to ensure it works correctly in all cases w/o causing
significant issues.
> fsck -includeSnapshots reports wrong amount of total blocks
> ---
>
> Key: HDFS-12618
> URL: https://issues.apache.org/jira/browse/HDFS-12618
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Affects Versions: 3.0.0-alpha3
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-121618.initial, HDFS-12618.001.patch,
> HDFS-12618.002.patch
>
>
> When snapshot is enabled, if a file is deleted but is contained by a
> snapshot, *fsck* will not reported blocks for such file, showing different
> number of *total blocks* than what is exposed in the Web UI.
> This should be fine, as *fsck* provides *-includeSnapshots* option. The
> problem is that *-includeSnapshots* option causes *fsck* to count blocks for
> every occurrence of a file on snapshots, which is wrong because these blocks
> should be counted only once (for instance, if a 100MB file is present on 3
> snapshots, it would still map to one block only in hdfs). This causes fsck to
> report much more blocks than what actually exist in hdfs and is reported in
> the Web UI.
> Here's an example:
> 1) HDFS has two files of 2 blocks each:
> {noformat}
> $ hdfs dfs -ls -R /
> drwxr-xr-x - root supergroup 0 2017-10-07 21:21 /snap-test
> -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:16 /snap-test/file1
> -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:17 /snap-test/file2
> drwxr-xr-x - root supergroup 0 2017-05-13 13:03 /test
> {noformat}
> 2) There are two snapshots, with the two files present on each of the
> snapshots:
> {noformat}
> $ hdfs dfs -ls -R /snap-test/.snapshot
> drwxr-xr-x - root supergroup 0 2017-10-07 21:21
> /snap-test/.snapshot/snap1
> -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:16
> /snap-test/.snapshot/snap1/file1
> -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:17
> /snap-test/.snapshot/snap1/file2
> drwxr-xr-x - root supergroup 0 2017-10-07 21:21
> /snap-test/.snapshot/snap2
> -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:16
> /snap-test/.snapshot/snap2/file1
> -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:17
> /snap-test/.snapshot/snap2/file2
> {noformat}
> 3) *fsck -includeSnapshots* reports 12 blocks in total (4 blocks for the
> normal file path, plus 4 blocks for each snapshot path):
> {noformat}
> $ hdfs fsck / -includeSnapshots
> FSCK started by root (auth:SIMPLE) from /127.0.0.1 for path / at Mon Oct 09
> 15:15:36 BST 2017
> Status: HEALTHY
> Number of data-nodes:1
>
[jira] [Updated] (HDFS-12396) Webhdfs file system should get delegation token from kms provider.
[ https://issues.apache.org/jira/browse/HDFS-12396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HDFS-12396: -- Status: Open (was: Patch Available) Canceling the patch to address Daryn's comments. > Webhdfs file system should get delegation token from kms provider. > -- > > Key: HDFS-12396 > URL: https://issues.apache.org/jira/browse/HDFS-12396 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: encryption, kms, webhdfs >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > Attachments: HDFS-12396.001.patch, HDFS-12396.002.patch, > HDFS-12396.003.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12678) Ozone: Corona: Add statistical information to json output
[ https://issues.apache.org/jira/browse/HDFS-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDFS-12678: --- Attachment: HDFS-12678-HDFS-7240.003.patch Thanks for the review [~nandakumar131]. v3 patch addresses your comments. Regarding the quantile, 11 does give us 11 cutpoints but the first cutpoint is called the zeroth quantile. > Ozone: Corona: Add statistical information to json output > - > > Key: HDFS-12678 > URL: https://issues.apache.org/jira/browse/HDFS-12678 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Lokesh Jain >Assignee: Lokesh Jain > Attachments: HDFS-12678-HDFS-7240.001.patch, > HDFS-12678-HDFS-7240.002.patch, HDFS-12678-HDFS-7240.003.patch > > > The Corona json output should contain statistical information like mean, > quantiles and standard deviation of time taken by ozone operations. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Closed] (HDFS-12715) I didn't find the "-find" command on hadoop2.6
[ https://issues.apache.org/jira/browse/HDFS-12715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal closed HDFS-12715. > I didn't find the "-find" command on hadoop2.6 > -- > > Key: HDFS-12715 > URL: https://issues.apache.org/jira/browse/HDFS-12715 > Project: Hadoop HDFS > Issue Type: Bug > Components: auto-failover >Affects Versions: 2.6.0 >Reporter: Djc >Priority: Critical > > When I looked for files on HDFS, I found no "find" command. I didn't find the > "find" command by using "Hadoop FS -help". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12702) Ozone: Add hugo to the dev docker image
[
https://issues.apache.org/jira/browse/HDFS-12702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220792#comment-16220792
]
Anu Engineer commented on HDFS-12702:
-
+1, I will commit this shortly.
> Ozone: Add hugo to the dev docker image
> ---
>
> Key: HDFS-12702
> URL: https://issues.apache.org/jira/browse/HDFS-12702
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
> Attachments: HDFS-12702-HDFS-7240.001.patch,
> HDFS-12702-HDFS-7240.002.patch
>
>
> Both HADOOP-14163 and HDFS-12664 requries hugo site generation tool. To make
> it easier to review those patches I suggest to add Hugo to the dev docker
> image now.
> This patch adds hugo to the dev docker image:
> Test method:
> {code}
> cd dev-support/docker
> docker build -t test .
> docker run test hugo version
> docker rmi test
> {code}
> Expected output (after docker run):
> {code}
> Hugo Static Site Generator v0.30.2 linux/amd64 BuildDate: 2017-10-19T11:34:27Z
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12693) Ozone: Enable XFrame options for KSM/SCM web ui
[
https://issues.apache.org/jira/browse/HDFS-12693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220802#comment-16220802
]
Anu Engineer commented on HDFS-12693:
-
+1, I will commit this shortly.
> Ozone: Enable XFrame options for KSM/SCM web ui
> ---
>
> Key: HDFS-12693
> URL: https://issues.apache.org/jira/browse/HDFS-12693
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
> Attachments: HDFS-12693-HDFS-7240.001.patch
>
>
> According to the discussion about security checklist on dev list I started to
> check the security features of the existing HttpServer2 and found that by
> default the XFrame option headers are disabled. This patch enables it by
> default for SCM/KSM server similar to the Namenode/Datanode webui.
> (Note: Even if the only form on the SCM/KSM ui-s is the standard LogLevel
> form, I think it's a good practice to enable it by default.)
> Test:
> Without the patch (clean build, SCM ui):
> {code}
> curl -v localhost:9876/jmx -o /dev/null
>
>* TCP_NODELAY set
> * Connected to localhost (::1) port 9876 (#0)
> > GET /jmx HTTP/1.1
> > Host: localhost:9876
> > User-Agent: curl/7.55.1
> > Accept: */*
> >
> < HTTP/1.1 200 OK
> < Date: Sat, 21 Oct 2017 19:54:43 GMT
> < Cache-Control: no-cache
> < Expires: Sat, 21 Oct 2017 19:54:43 GMT
> < Date: Sat, 21 Oct 2017 19:54:43 GMT
> < Pragma: no-cache
> < Content-Type: application/json; charset=utf8
> < Access-Control-Allow-Methods: GET
> < Access-Control-Allow-Origin: *
> < Transfer-Encoding: chunked
> {code}
> With the patch:
> {code}
> curl -v localhost:9876/jmx -o /dev/null
>
> * Connected to localhost (::1) port 9876 (#0)
> > GET /jmx HTTP/1.1
> > Host: localhost:9876
> > User-Agent: curl/7.55.1
> > Accept: */*
> >
> < HTTP/1.1 200 OK
> < Date: Sat, 21 Oct 2017 19:55:07 GMT
> < Cache-Control: no-cache
> < Expires: Sat, 21 Oct 2017 19:55:07 GMT
> < Date: Sat, 21 Oct 2017 19:55:07 GMT
> < Pragma: no-cache
> < Content-Type: application/json; charset=utf8
> < X-FRAME-OPTIONS: SAMEORIGIN
> < Access-Control-Allow-Methods: GET
> < Access-Control-Allow-Origin: *
> < Transfer-Encoding: chunked
> {code}
> Note: X-FRAME-OPTIONS header exists at the second case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Status: Open (was: Patch Available)
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Attachment: (was: HDFS-11902-HDFS-9806.010.patch)
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Attachment: HDFS-11902-HDFS-9806.010.patch
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12690) Ozone: generate swagger descriptor for the Ozone REST Api
[ https://issues.apache.org/jira/browse/HDFS-12690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220828#comment-16220828 ] Anu Engineer commented on HDFS-12690: - +1, I am will commit this shortly. > Ozone: generate swagger descriptor for the Ozone REST Api > - > > Key: HDFS-12690 > URL: https://issues.apache.org/jira/browse/HDFS-12690 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Attachments: HDFS-12690-HDFS-7240.001.patch > > > This patch generates ozone.swagger.json descriptor during compilation time > and adds it to the static/ web folder (available both from all web ui). > Note: I tested multiple method to generate swagger file: runtime and > compilation time. Runtime needs more additional dependencies and multiple > workaround as we have no ServletContext and ServletConfig in the netty based > adapter (but it's possible to add a stub one). I prefer the compilation time > generation because it more simplified and it also could be used to generate > additional documentation. > This patch contains only the basic @Api/@ApiMethod annotations the parameters > not yet annotated. Followup tasks: > * We can check how can we generate parameter level description from the > javadoc. It is possible with custom docket + custom swagger reader, but maybe > doesn't worth it. > * We can add a swagger ui (there are many implementations). It's a licence > nightmare as most of the ui contains unlimited number of npm dependency with > different (but mostly Apache + MIT) artifacts. I will suggest to add a > swagger ui without the javascript and load it from cdn. It will work only > with active internet connection but without licencing issue. > * Long term with this plugin we can also generate the content of > OzoneRest.md (after a fine tuning the swagger annotations) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9914) Fix configurable WebhDFS connect/read timeout
[ https://issues.apache.org/jira/browse/HDFS-9914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220858#comment-16220858 ] Arpit Agarwal commented on HDFS-9914: - Thanks [~xyao]. +1 for the v2 patch, pending Jenkins. I've kicked off a new pre-commit run for this issue. > Fix configurable WebhDFS connect/read timeout > - > > Key: HDFS-9914 > URL: https://issues.apache.org/jira/browse/HDFS-9914 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Blocker > Fix For: 3.0.0 > > Attachments: HDFS-9914.001.patch, HDFS-9914.002.patch > > > Webhdfs specific read/connect timeout as added HDFS-9887. This ticket is > opened to fix the following issues in current implementation: > 1. The webhdfs read/connect timeout should not affect connection for other > callers of URLConnectionFactory.newSslConnConfigurator() such as > QuorumJournalManager#QuorumJournalManger(), DFSck#DFSck() and > TransferFsImage() > 2. URLConnectionFactory#getSSLConnectionConfiguration() should honor webhdfs > connect/read timeout even if any exception is thrown during customized SSL > configuration. > > 3. OAuth2 webhdfs connection should honor the webhdfs connect/read timeout. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12582) Replace HdfsFileStatus constructor with a builder pattern.
[ https://issues.apache.org/jira/browse/HDFS-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220883#comment-16220883 ] Bharat Viswanadham commented on HDFS-12582: --- [~chris.douglas] I am okay with going this change as part of HDFS-12681. > Replace HdfsFileStatus constructor with a builder pattern. > -- > > Key: HDFS-12582 > URL: https://issues.apache.org/jira/browse/HDFS-12582 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham > Attachments: HDFS-12582.01.patch, HDFS-12582.02.patch, > HDFS-12582.03.patch, HDFS-12582.04.patch, HDFS-12582.05.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7878) API - expose an unique file identifier
[
https://issues.apache.org/jira/browse/HDFS-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220898#comment-16220898
]
Sean Mackrory commented on HDFS-7878:
-
{quote}Wire compatibility with HDFS{quote}
How do commented out lines ensure wire compatibility? It would make sense if
these were obsolete fields and we didn't want to reuse obsolete number in case
older messages get misinterpreted, but then we should be reusing. Nevertheless,
it appears we're not doing that in the latest patch anymore.
I do think this resolves my previous concerns with the patch. In
testCrossSerializationProto and testJavaSerialization we're removing assertions
that the PathHandle to what should be the same file should be identical. Isn't
that still true, and should be?
> API - expose an unique file identifier
> --
>
> Key: HDFS-7878
> URL: https://issues.apache.org/jira/browse/HDFS-7878
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Labels: BB2015-05-TBR
> Attachments: HDFS-7878.01.patch, HDFS-7878.02.patch,
> HDFS-7878.03.patch, HDFS-7878.04.patch, HDFS-7878.05.patch,
> HDFS-7878.06.patch, HDFS-7878.07.patch, HDFS-7878.08.patch,
> HDFS-7878.09.patch, HDFS-7878.10.patch, HDFS-7878.11.patch,
> HDFS-7878.12.patch, HDFS-7878.13.patch, HDFS-7878.14.patch,
> HDFS-7878.15.patch, HDFS-7878.16.patch, HDFS-7878.17.patch,
> HDFS-7878.18.patch, HDFS-7878.patch
>
>
> See HDFS-487.
> Even though that is resolved as duplicate, the ID is actually not exposed by
> the JIRA it supposedly duplicates.
> INode ID for the file should be easy to expose; alternatively ID could be
> derived from block IDs, to account for appends...
> This is useful e.g. for cache key by file, to make sure cache stays correct
> when file is overwritten.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12310) [SPS]: Provide an option to track the status of in progress requests
[ https://issues.apache.org/jira/browse/HDFS-12310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220901#comment-16220901 ] Rakesh R commented on HDFS-12310: - Thanks [~surendrasingh] for the quick updates. The new patch looks good to me. [~eddyxu], would be great to see your feedback on the latest patch. I will wait before committing this. Thanks! > [SPS]: Provide an option to track the status of in progress requests > > > Key: HDFS-12310 > URL: https://issues.apache.org/jira/browse/HDFS-12310 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G >Assignee: Surendra Singh Lilhore > Attachments: HDFS-12310-HDFS-10285-01.patch, > HDFS-12310-HDFS-10285-02.patch, HDFS-12310-HDFS-10285-03.patch, > HDFS-12310-HDFS-10285-04.patch, HDFS-12310-HDFS-10285-05.patch > > > As per the [~andrew.wang] 's review comments in HDFS-10285, This is the JIRA > for tracking about the options how we track the progress of SPS requests. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10285) Storage Policy Satisfier in Namenode
[ https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220907#comment-16220907 ] Daryn Sharp commented on HDFS-10285: I see a few issues here but in a fire fight. Stay tuned. > Storage Policy Satisfier in Namenode > > > Key: HDFS-10285 > URL: https://issues.apache.org/jira/browse/HDFS-10285 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10285-consolidated-merge-patch-00.patch, > HDFS-10285-consolidated-merge-patch-01.patch, > HDFS-SPS-TestReport-20170708.pdf, > Storage-Policy-Satisfier-in-HDFS-June-20-2017.pdf, > Storage-Policy-Satisfier-in-HDFS-May10.pdf, > Storage-Policy-Satisfier-in-HDFS-Oct-26-2017.pdf > > > Heterogeneous storage in HDFS introduced the concept of storage policy. These > policies can be set on directory/file to specify the user preference, where > to store the physical block. When user set the storage policy before writing > data, then the blocks could take advantage of storage policy preferences and > stores physical block accordingly. > If user set the storage policy after writing and completing the file, then > the blocks would have been written with default storage policy (nothing but > DISK). User has to run the ‘Mover tool’ explicitly by specifying all such > file names as a list. In some distributed system scenarios (ex: HBase) it > would be difficult to collect all the files and run the tool as different > nodes can write files separately and file can have different paths. > Another scenarios is, when user rename the files from one effected storage > policy file (inherited policy from parent directory) to another storage > policy effected directory, it will not copy inherited storage policy from > source. So it will take effect from destination file/dir parent storage > policy. This rename operation is just a metadata change in Namenode. The > physical blocks still remain with source storage policy. > So, Tracking all such business logic based file names could be difficult for > admins from distributed nodes(ex: region servers) and running the Mover tool. > Here the proposal is to provide an API from Namenode itself for trigger the > storage policy satisfaction. A Daemon thread inside Namenode should track > such calls and process to DN as movement commands. > Will post the detailed design thoughts document soon. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12655) Ozone: use specific names for RPC.Server instances
[ https://issues.apache.org/jira/browse/HDFS-12655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-12655: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-7240 Status: Resolved (was: Patch Available) [~elek] Thanks for the contribution. I have committed this to the feature branch > Ozone: use specific names for RPC.Server instances > -- > > Key: HDFS-12655 > URL: https://issues.apache.org/jira/browse/HDFS-12655 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Fix For: HDFS-7240 > > Attachments: HDFS-12655-HDFS-7240.001.patch, rpcmetrics.png > > > My original motivation is using meaningful names on the scm web ui. As you > can see on the attached screenshot we display the metrics for all the RPC > servers (using hadoop metrics published on jmx). Unfortunatelly we can > display only the port number on the web ui as there are no meaningfull names > published on the jmx interface. > After some investigation I found that there is a serverName constructor > parameter int Rpc.Server but it is NOT used currently. > This patch will: > 1. store the serverName in field of RPC.Server a > 2. Improve the way how the serverName is calculated from the protocol class > names (it's typically an anonnymous inner class, so I remove the unnecessary > $-s from the name.) > 3. Add a new tag to the RpcMetrics based on the serverName field of the > Rpc.Server It will be displayed over JMX > 4. Add unit tests for checking the tag values and the default > classname->servername mapping. > ps: I need it for Ozone SCM web ui, but let ne know if it should be moved to > HADOOP project/trunk. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12558) Ozone: Clarify the meaning of rpc.metrics.percentiles.intervals on KSM/SCM web ui
[ https://issues.apache.org/jira/browse/HDFS-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-12558: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-7240 Status: Resolved (was: Patch Available) [~elek] Thanks for the contribution. I have committed this to the feature branch. > Ozone: Clarify the meaning of rpc.metrics.percentiles.intervals on KSM/SCM > web ui > - > > Key: HDFS-12558 > URL: https://issues.apache.org/jira/browse/HDFS-12558 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Fix For: HDFS-7240 > > Attachments: HDFS-12558-HDFS-7240.001.patch, > HDFS-12558-HDFS-7240.002.patch, after.png, before.png > > > In Ozone (SCM/KSM) web ui we have additional visualization if > rpc.metrics.percentiles.intervals are enabled. > But according to the feedbacks it's a little bit confusing what is it exactly. > I would like to improve it and clarify how does it work. > 1. I will to add a footnote about these are not rolling windows but just > display of the last fixed window. > 2. I would like to rearrange the layout. As the different windows are > independent, I would show them in different lines and group by the intervals > and not by RpcQueueTime/RpcProcessingTime. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12693) Ozone: Enable XFrame options for KSM/SCM web ui
[
https://issues.apache.org/jira/browse/HDFS-12693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-12693:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HDFS-7240
Status: Resolved (was: Patch Available)
[~elek] Thanks for the contribution. I have committed this to the feature branch
> Ozone: Enable XFrame options for KSM/SCM web ui
> ---
>
> Key: HDFS-12693
> URL: https://issues.apache.org/jira/browse/HDFS-12693
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
> Fix For: HDFS-7240
>
> Attachments: HDFS-12693-HDFS-7240.001.patch
>
>
> According to the discussion about security checklist on dev list I started to
> check the security features of the existing HttpServer2 and found that by
> default the XFrame option headers are disabled. This patch enables it by
> default for SCM/KSM server similar to the Namenode/Datanode webui.
> (Note: Even if the only form on the SCM/KSM ui-s is the standard LogLevel
> form, I think it's a good practice to enable it by default.)
> Test:
> Without the patch (clean build, SCM ui):
> {code}
> curl -v localhost:9876/jmx -o /dev/null
>
>* TCP_NODELAY set
> * Connected to localhost (::1) port 9876 (#0)
> > GET /jmx HTTP/1.1
> > Host: localhost:9876
> > User-Agent: curl/7.55.1
> > Accept: */*
> >
> < HTTP/1.1 200 OK
> < Date: Sat, 21 Oct 2017 19:54:43 GMT
> < Cache-Control: no-cache
> < Expires: Sat, 21 Oct 2017 19:54:43 GMT
> < Date: Sat, 21 Oct 2017 19:54:43 GMT
> < Pragma: no-cache
> < Content-Type: application/json; charset=utf8
> < Access-Control-Allow-Methods: GET
> < Access-Control-Allow-Origin: *
> < Transfer-Encoding: chunked
> {code}
> With the patch:
> {code}
> curl -v localhost:9876/jmx -o /dev/null
>
> * Connected to localhost (::1) port 9876 (#0)
> > GET /jmx HTTP/1.1
> > Host: localhost:9876
> > User-Agent: curl/7.55.1
> > Accept: */*
> >
> < HTTP/1.1 200 OK
> < Date: Sat, 21 Oct 2017 19:55:07 GMT
> < Cache-Control: no-cache
> < Expires: Sat, 21 Oct 2017 19:55:07 GMT
> < Date: Sat, 21 Oct 2017 19:55:07 GMT
> < Pragma: no-cache
> < Content-Type: application/json; charset=utf8
> < X-FRAME-OPTIONS: SAMEORIGIN
> < Access-Control-Allow-Methods: GET
> < Access-Control-Allow-Origin: *
> < Transfer-Encoding: chunked
> {code}
> Note: X-FRAME-OPTIONS header exists at the second case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12702) Ozone: Add hugo to the dev docker image
[
https://issues.apache.org/jira/browse/HDFS-12702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-12702:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HDFS-7240
Status: Resolved (was: Patch Available)
[~elek] Thanks for the contribution. I have committed this to the feature
branch.
> Ozone: Add hugo to the dev docker image
> ---
>
> Key: HDFS-12702
> URL: https://issues.apache.org/jira/browse/HDFS-12702
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
> Fix For: HDFS-7240
>
> Attachments: HDFS-12702-HDFS-7240.001.patch,
> HDFS-12702-HDFS-7240.002.patch
>
>
> Both HADOOP-14163 and HDFS-12664 requries hugo site generation tool. To make
> it easier to review those patches I suggest to add Hugo to the dev docker
> image now.
> This patch adds hugo to the dev docker image:
> Test method:
> {code}
> cd dev-support/docker
> docker build -t test .
> docker run test hugo version
> docker rmi test
> {code}
> Expected output (after docker run):
> {code}
> Hugo Static Site Generator v0.30.2 linux/amd64 BuildDate: 2017-10-19T11:34:27Z
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12106) [SPS]: Improve storage policy satisfier configurations
[
https://issues.apache.org/jira/browse/HDFS-12106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220949#comment-16220949
]
Rakesh R commented on HDFS-12106:
-
Thanks [~surendrasingh] for the patch. Adding few comments, please take care.
# HDFS-12310 jira proposing {{-w}} option and the cmd will indefinitely
waiting to finish all the blocks movement, then giving a SUCCESS status. But
with this proposed patch it is breaking the indefinite loop and exiting with
new final state - failure. Here, we could add a new status
{{StoragePolicySatisfyPathStatus#FAILURE}}, this represents that 'all blocks'
or 'few blocks' failed to do the block movement and the path is still not fully
satisfied the storage policy. This will allow the users to take a call retry by
themselves.
# Please uncomment {{ @Test//(timeout = 30)}}
# Add the config {{dfs.storage.policy.satisfier.max.retry.count}} to
hdfs-default.xml file
# As we are changing the default to {{true}}, could you update the doc as well
{code}
dfs.storage.policy.satisfier.low.max-streams.preference
false
If true, blocks to move tasks will share equal ratio of number of
highest-priority
replication streams (dfs.namenode.replication.max-streams) with pending
replica and
erasure-coded reconstruction tasks. If false, blocks to move tasks will
only use
the delta number of replication streams. The default value is false.
{code}
> [SPS]: Improve storage policy satisfier configurations
> --
>
> Key: HDFS-12106
> URL: https://issues.apache.org/jira/browse/HDFS-12106
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12106-HDFS-10285-01.patch
>
>
> Following are the changes doing as part of this task:-
> # Make satisfy policy retry configurable.
> Based on
> [discussion|https://issues.apache.org/jira/browse/HDFS-11965?focusedCommentId=16074338&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16074338]
> in HDFS-11965, we can make satisfy policy retry configurable.
> # Change {{dfs.storage.policy.satisfier.low.max-streams.preference}}'s value
> to {{true}} and modify the default value to true as well. If user wants equal
> share then it should be false, but presently it is true which is not correct.
> Thanks [~umamaheswararao] for pointing out this case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10285) Storage Policy Satisfier in Namenode
[ https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220974#comment-16220974 ] Uma Maheswara Rao G commented on HDFS-10285: Hi [~daryn], great to see you are getting some time on this. Thanks a lot for your reviews. We will wait for your comments. > Storage Policy Satisfier in Namenode > > > Key: HDFS-10285 > URL: https://issues.apache.org/jira/browse/HDFS-10285 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10285-consolidated-merge-patch-00.patch, > HDFS-10285-consolidated-merge-patch-01.patch, > HDFS-SPS-TestReport-20170708.pdf, > Storage-Policy-Satisfier-in-HDFS-June-20-2017.pdf, > Storage-Policy-Satisfier-in-HDFS-May10.pdf, > Storage-Policy-Satisfier-in-HDFS-Oct-26-2017.pdf > > > Heterogeneous storage in HDFS introduced the concept of storage policy. These > policies can be set on directory/file to specify the user preference, where > to store the physical block. When user set the storage policy before writing > data, then the blocks could take advantage of storage policy preferences and > stores physical block accordingly. > If user set the storage policy after writing and completing the file, then > the blocks would have been written with default storage policy (nothing but > DISK). User has to run the ‘Mover tool’ explicitly by specifying all such > file names as a list. In some distributed system scenarios (ex: HBase) it > would be difficult to collect all the files and run the tool as different > nodes can write files separately and file can have different paths. > Another scenarios is, when user rename the files from one effected storage > policy file (inherited policy from parent directory) to another storage > policy effected directory, it will not copy inherited storage policy from > source. So it will take effect from destination file/dir parent storage > policy. This rename operation is just a metadata change in Namenode. The > physical blocks still remain with source storage policy. > So, Tracking all such business logic based file names could be difficult for > admins from distributed nodes(ex: region servers) and running the Mover tool. > Here the proposal is to provide an API from Namenode itself for trigger the > storage policy satisfaction. A Daemon thread inside Namenode should track > such calls and process to DN as movement commands. > Will post the detailed design thoughts document soon. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12690) Ozone: generate swagger descriptor for the Ozone REST Api
[ https://issues.apache.org/jira/browse/HDFS-12690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-12690: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-7240 Status: Resolved (was: Patch Available) [~elek] Thanks for the contribution. I have committed this to the feature branch. > Ozone: generate swagger descriptor for the Ozone REST Api > - > > Key: HDFS-12690 > URL: https://issues.apache.org/jira/browse/HDFS-12690 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Fix For: HDFS-7240 > > Attachments: HDFS-12690-HDFS-7240.001.patch > > > This patch generates ozone.swagger.json descriptor during compilation time > and adds it to the static/ web folder (available both from all web ui). > Note: I tested multiple method to generate swagger file: runtime and > compilation time. Runtime needs more additional dependencies and multiple > workaround as we have no ServletContext and ServletConfig in the netty based > adapter (but it's possible to add a stub one). I prefer the compilation time > generation because it more simplified and it also could be used to generate > additional documentation. > This patch contains only the basic @Api/@ApiMethod annotations the parameters > not yet annotated. Followup tasks: > * We can check how can we generate parameter level description from the > javadoc. It is possible with custom docket + custom swagger reader, but maybe > doesn't worth it. > * We can add a swagger ui (there are many implementations). It's a licence > nightmare as most of the ui contains unlimited number of npm dependency with > different (but mostly Apache + MIT) artifacts. I will suggest to add a > swagger ui without the javascript and load it from cdn. It will work only > with active internet connection but without licencing issue. > * Long term with this plugin we can also generate the content of > OzoneRest.md (after a fine tuning the swagger annotations) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12678) Ozone: Corona: Add statistical information to json output
[ https://issues.apache.org/jira/browse/HDFS-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221011#comment-16221011 ] Nanda kumar commented on HDFS-12678: Thanks [~ljain] for the update. +1, the patch looks good to me. I will commit this shortly. > Ozone: Corona: Add statistical information to json output > - > > Key: HDFS-12678 > URL: https://issues.apache.org/jira/browse/HDFS-12678 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Lokesh Jain >Assignee: Lokesh Jain > Attachments: HDFS-12678-HDFS-7240.001.patch, > HDFS-12678-HDFS-7240.002.patch, HDFS-12678-HDFS-7240.003.patch > > > The Corona json output should contain statistical information like mean, > quantiles and standard deviation of time taken by ozone operations. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12680) Ozone: SCM: Lease support for container creation
[
https://issues.apache.org/jira/browse/HDFS-12680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221015#comment-16221015
]
Anu Engineer commented on HDFS-12680:
-
+1, I will commit this shortly.
[~xyao] I am aware that the state machine inside the {{ContainerStateMachine}}
is not exactly what we need, It will be fixed in the next JIRA that uses Lease
Manager for pipelines.
[~nandakumar131] Could you please file JIRAs for both -- moving the Ratis
pipeline create to client side and adding lease manager to the pipeline create
path.
> Ozone: SCM: Lease support for container creation
>
>
> Key: HDFS-12680
> URL: https://issues.apache.org/jira/browse/HDFS-12680
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nanda kumar
>Assignee: Nanda kumar
> Labels: ozoneMerge
> Attachments: HDFS-12680-HDFS-7240.000.patch,
> HDFS-12680-HDFS-7240.001.patch
>
>
> This brings in lease support for container creation.
> Lease should be give for a container that is moved to {{CREATING}} state when
> {{BEGIN_CREATE}} event happens, {{LeaseException}} should be thrown if the
> container already holds a lease. Lease must be released during
> {{COMPLETE_CREATE}} event. If the lease times out container should be moved
> to {{DELETING}} state, and exception should be thrown if {{COMPLETE_CREATE}}
> event is received on that container.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12680) Ozone: SCM: Lease support for container creation
[
https://issues.apache.org/jira/browse/HDFS-12680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-12680:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HDFS-7240
Status: Resolved (was: Patch Available)
[~linyiqun] Thanks for the reviews.
[~nandakumar131] Thanks for the contribution. I have committed this to the
feature branch. I have run all ozone tests on my machines and verified that the
latest patch works correctly.
> Ozone: SCM: Lease support for container creation
>
>
> Key: HDFS-12680
> URL: https://issues.apache.org/jira/browse/HDFS-12680
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nanda kumar
>Assignee: Nanda kumar
> Labels: ozoneMerge
> Fix For: HDFS-7240
>
> Attachments: HDFS-12680-HDFS-7240.000.patch,
> HDFS-12680-HDFS-7240.001.patch
>
>
> This brings in lease support for container creation.
> Lease should be give for a container that is moved to {{CREATING}} state when
> {{BEGIN_CREATE}} event happens, {{LeaseException}} should be thrown if the
> container already holds a lease. Lease must be released during
> {{COMPLETE_CREATE}} event. If the lease times out container should be moved
> to {{DELETING}} state, and exception should be thrown if {{COMPLETE_CREATE}}
> event is received on that container.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12678) Ozone: Corona: Add statistical information to json output
[ https://issues.apache.org/jira/browse/HDFS-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nanda kumar updated HDFS-12678: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-7240 Status: Resolved (was: Patch Available) [~ljain] Thanks for the contribution. I have committed this to the feature branch. > Ozone: Corona: Add statistical information to json output > - > > Key: HDFS-12678 > URL: https://issues.apache.org/jira/browse/HDFS-12678 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Lokesh Jain >Assignee: Lokesh Jain > Fix For: HDFS-7240 > > Attachments: HDFS-12678-HDFS-7240.001.patch, > HDFS-12678-HDFS-7240.002.patch, HDFS-12678-HDFS-7240.003.patch > > > The Corona json output should contain statistical information like mean, > quantiles and standard deviation of time taken by ozone operations. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12711) deadly hdfs test
[
https://issues.apache.org/jira/browse/HDFS-12711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221069#comment-16221069
]
Allen Wittenauer commented on HDFS-12711:
-
https://builds.apache.org/job/PreCommit-HDFS-Build2/2
This is better or worse, depending upon your point of view.
Many of these:
{code}
estDeleteEZWithMultipleUsers(org.apache.hadoop.hdfs.TestTrashWithEncryptionZones)
Time elapsed: 5.134 sec <<< ERROR!
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at
io.netty.util.concurrent.SingleThreadEventExecutor.shutdownGracefully(SingleThreadEventExecutor.java:557)
at
io.netty.util.concurrent.MultithreadEventExecutorGroup.shutdownGracefully(MultithreadEventExecutorGroup.java:146)
at
io.netty.util.concurrent.AbstractEventExecutorGroup.shutdownGracefully(AbstractEventExecutorGroup.java:69)
at
org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer.close(DatanodeHttpServer.java:272)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:1986)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNode(MiniDFSCluster.java:1868)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNodes(MiniDFSCluster.java:1858)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1837)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1811)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1804)
at
org.apache.hadoop.hdfs.TestTrashWithEncryptionZones.teardown(TestTrashWithEncryptionZones.java:118)
{code}
which eventually turn into these:
{code}
/bin/sh: 1: Cannot fork
/bin/sh: 1: Cannot fork
/bin/sh: 1: Cannot fork
/bin/sh: 1: Cannot fork
/bin/sh: 1: Cannot fork
/bin/sh: 1: Cannot fork
{code}
but that's not it's final form! Oh no, it then turns into reams and reams of
this:
{code}
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# An error report file with more information is saved as:
# /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/hs_err_pid20306.log
#
{code}
Now we just have to wait and see if the walls put up were enough to stop H4
from crashing.
(the 1k default process limit is *probably* a smidge too low, but not by too
much.)
> deadly hdfs test
>
>
> Key: HDFS-12711
> URL: https://issues.apache.org/jira/browse/HDFS-12711
> Project: Hadoop HDFS
> Issue Type: Test
>Affects Versions: 2.9.0, 2.8.2
>Reporter: Allen Wittenauer
>Priority: Critical
> Attachments: HDFS-12711.branch-2.00.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12718) Ozone: fix thread number calculation in CBlockManager
[
https://issues.apache.org/jira/browse/HDFS-12718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-12718:
-
Parent Issue: HDFS-8 (was: HDFS-7240)
> Ozone: fix thread number calculation in CBlockManager
> -
>
> Key: HDFS-12718
> URL: https://issues.apache.org/jira/browse/HDFS-12718
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
> Attachments: HDFS-12718-HDFS-7240.001.patch
>
>
> When starting cblock server or during the unit tests I got many
> IllegalArgumentException:
> {code}
> testCliInfoVolume(org.apache.hadoop.cblock.TestCBlockCLI) Time elapsed:
> 0.004 sec <<< ERROR!
> org.apache.hadoop.ipc.RemoteException: java.lang.IllegalArgumentException
> at
> java.util.concurrent.ThreadPoolExecutor.(ThreadPoolExecutor.java:1307)
> at
> java.util.concurrent.ThreadPoolExecutor.(ThreadPoolExecutor.java:1265)
> at
> org.apache.hadoop.cblock.storage.StorageManager.createVolumeContainers(StorageManager.java:212)
> at
> org.apache.hadoop.cblock.storage.StorageManager.createVolume(StorageManager.java:304)
> at
> org.apache.hadoop.cblock.CBlockManager.createVolume(CBlockManager.java:257)
> at
> org.apache.hadoop.cblock.protocolPB.CBlockServiceProtocolServerSideTranslatorPB.createVolume(CBlockServiceProtocolServerSideTranslatorPB.java:57)
> at
> org.apache.hadoop.cblock.protocol.proto.CBlockServiceProtocolProtos$CBlockServiceProtocolService$2.callBlockingMethod(CBlockServiceProtocolProtos.java:6056)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491)
> at org.apache.hadoop.ipc.Client.call(Client.java:1437)
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy10.createVolume(Unknown Source)
> at
> org.apache.hadoop.cblock.client.CBlockServiceProtocolClientSideTranslatorPB.createVolume(CBlockServiceProtocolClientSideTranslatorPB.java:64)
> at
> org.apache.hadoop.cblock.client.CBlockVolumeClient.createVolume(CBlockVolumeClient.java:64)
> at
> org.apache.hadoop.cblock.cli.CBlockCli.createVolume(CBlockCli.java:239)
> at org.apache.hadoop.cblock.cli.CBlockCli.run(CBlockCli.java:173)
> at
> org.apache.hadoop.cblock.TestCBlockCLI.testCliInfoVolume(TestCBlockCLI.java:232)
> {code}
> The root cause is that in CBlock Manager we create ThreadGroups:
> {code}
> ThreadPoolExecutor executor = new ThreadPoolExecutor(numThreads,
> MAX_THREADS, 1, TimeUnit.SECONDS,
> new ArrayBlockingQueue<>(MAX_QUEUE_CAPACITY),
> new ThreadPoolExecutor.CallerRunsPolicy());
> {code}
> Where numThreads (the number of always active threads) comes from config and
> 16 by default MAX_THREADS is `Runtime.getRuntime().availableProcessors() * 2`.
> My problem was that MAX_THREAD was lower than numThreads (as I have only 2
> processors, shame on me), so I got IllegalArgumentException.
> In the fix I suggest:
> * Limit the maximum number of threads not the always active threads, as
> ususally this is the number which is needed to adjust
> * Use the core dependent numbers as the number of active threas (but if
> numThreads is smaller, that should be used).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12719) Ozone: Fix checkstyle, javac, whitespace issues in HDFS-7240 branch
Mukul Kumar Singh created HDFS-12719: Summary: Ozone: Fix checkstyle, javac, whitespace issues in HDFS-7240 branch Key: HDFS-12719 URL: https://issues.apache.org/jira/browse/HDFS-12719 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Mukul Kumar Singh Assignee: Mukul Kumar Singh Fix For: HDFS-7240 There are outstanding whitespace/javac/checkstyle issues on the HDFS-7240 branch. These were observed by uploading the branch diff to the trunk via parent jira HDFS-7240. This jira will fix all the valid outstanding issues. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12711) deadly hdfs test
[
https://issues.apache.org/jira/browse/HDFS-12711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221076#comment-16221076
]
Hadoop QA commented on HDFS-12711:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
21s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 31m 7s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestFileStatus |
| | hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes |
| | hadoop.hdfs.web.TestWebHDFSAcl |
| | hadoop.hdfs.TestDFSMkdirs |
| | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.TestRollingUpgradeRollback |
| | hadoop.hdfs.TestTrashWithEncryptionZones |
| | hadoop.hdfs.web.TestFSMainOperationsWebHdfs |
| | hadoop.hdfs.TestBlockStoragePolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:17213a0 |
| JIRA Issue | HDFS-12711 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12893963/HDFS-12711.branch-2.00.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux 1ad1a0b77a83 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / 4678315 |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| Default Java | 1.7.0_151 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build2/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build2/2/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build2/2/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> deadly hdfs test
>
>
> Key: HDFS-12711
> URL: https://issues.apache.org/jira/browse/HDFS-12711
> Project: Hadoop HDFS
> Issue Type: Test
>Affects Versions: 2.9.0, 2.8.2
[jira] [Created] (HDFS-12720) Ozone: Ratis options are not passed from KSM Client protobuf helper correctly.
Mukul Kumar Singh created HDFS-12720:
Summary: Ozone: Ratis options are not passed from KSM Client
protobuf helper correctly.
Key: HDFS-12720
URL: https://issues.apache.org/jira/browse/HDFS-12720
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Affects Versions: HDFS-7240
Reporter: Mukul Kumar Singh
Assignee: Mukul Kumar Singh
Fix For: HDFS-7240
{{KeySpaceManagerProtocolClientSideTranslatorPB#allocateBlock}} and
{{KeySpaceManagerProtocolClientSideTranslatorPB#openKey}} do not pass the ratis
replication factor and replication type to the KSM server. this causes the
allocations using ratis model to resort to standalone mode even when Ratis mode
is specified.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12711) deadly hdfs test
[ https://issues.apache.org/jira/browse/HDFS-12711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221077#comment-16221077 ] Allen Wittenauer commented on HDFS-12711: - Success! One interesting thing: surefire is a liar. A lot more tests failed than what it reported. > deadly hdfs test > > > Key: HDFS-12711 > URL: https://issues.apache.org/jira/browse/HDFS-12711 > Project: Hadoop HDFS > Issue Type: Test >Affects Versions: 2.9.0, 2.8.2 >Reporter: Allen Wittenauer >Priority: Critical > Attachments: HDFS-12711.branch-2.00.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12721) Ozone: Ratis: Moving Ratis pipeline creation to client
Nanda kumar created HDFS-12721:
--
Summary: Ozone: Ratis: Moving Ratis pipeline creation to client
Key: HDFS-12721
URL: https://issues.apache.org/jira/browse/HDFS-12721
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Reporter: Nanda kumar
Currently after the allocation of pipeline SCM also creates the pipeline using
{{XceiverClientRatis}}. Creation of pipeline has to be moved to the client from
SCM. This is because the communication between Datanode and SCM happens only
through heartbeat (and container report), Datanode sends heartbeat to SCM and
SCM sends commands to Datanode through heartbeat response.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12711) deadly hdfs test
[ https://issues.apache.org/jira/browse/HDFS-12711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221083#comment-16221083 ] Allen Wittenauer commented on HDFS-12711: - Note to self: Kicking off another run with proclimit set to 1500. > deadly hdfs test > > > Key: HDFS-12711 > URL: https://issues.apache.org/jira/browse/HDFS-12711 > Project: Hadoop HDFS > Issue Type: Test >Affects Versions: 2.9.0, 2.8.2 >Reporter: Allen Wittenauer >Priority: Critical > Attachments: HDFS-12711.branch-2.00.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12663) Ozone: OzoneClient: Remove protobuf classes exposed to clients through OzoneBucket
[
https://issues.apache.org/jira/browse/HDFS-12663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-12663:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HDFS-7240
Status: Resolved (was: Patch Available)
[~nandakumar131] Thanks for the contribution. I have committed this to the
feature branch.
> Ozone: OzoneClient: Remove protobuf classes exposed to clients through
> OzoneBucket
> --
>
> Key: HDFS-12663
> URL: https://issues.apache.org/jira/browse/HDFS-12663
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nanda kumar
>Assignee: Nanda kumar
> Labels: ozoneMerge
> Fix For: HDFS-7240
>
> Attachments: HDFS-12663-HDFS-7240.000.patch,
> HDFS-12663-HDFS-7240.001.patch, HDFS-12663-HDFS-7240.002.patch
>
>
> As part of {{OzoneBucket#createKey}} we are currently exposing protobuf enums
> {{OzoneProtos.ReplicationType}} and {{OzoneProtos.ReplicationFactor}} through
> client, this can be replaced with client side enums and conversion can be
> done internally.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12722) Ozone: SCM: Lease support for pipeline creation
Nanda kumar created HDFS-12722: -- Summary: Ozone: SCM: Lease support for pipeline creation Key: HDFS-12722 URL: https://issues.apache.org/jira/browse/HDFS-12722 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Reporter: Nanda kumar This brings in lease support for pipeline creation. Whenever a new pipeline is allocated it is given to the client for creation, with lease we will avoid giving the same pipeline to more than one client for creation. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12720) Ozone: Ratis options are not passed from KSM Client protobuf helper correctly.
[
https://issues.apache.org/jira/browse/HDFS-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-12720:
-
Attachment: HDFS-12720-HDFS-7240.001.patch
> Ozone: Ratis options are not passed from KSM Client protobuf helper correctly.
> --
>
> Key: HDFS-12720
> URL: https://issues.apache.org/jira/browse/HDFS-12720
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: HDFS-7240
>
> Attachments: HDFS-12720-HDFS-7240.001.patch
>
>
> {{KeySpaceManagerProtocolClientSideTranslatorPB#allocateBlock}} and
> {{KeySpaceManagerProtocolClientSideTranslatorPB#openKey}} do not pass the
> ratis replication factor and replication type to the KSM server. this causes
> the allocations using ratis model to resort to standalone mode even when
> Ratis mode is specified.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12720) Ozone: Ratis options are not passed from KSM Client protobuf helper correctly.
[
https://issues.apache.org/jira/browse/HDFS-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-12720:
-
Status: Patch Available (was: Open)
> Ozone: Ratis options are not passed from KSM Client protobuf helper correctly.
> --
>
> Key: HDFS-12720
> URL: https://issues.apache.org/jira/browse/HDFS-12720
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: HDFS-7240
>
> Attachments: HDFS-12720-HDFS-7240.001.patch
>
>
> {{KeySpaceManagerProtocolClientSideTranslatorPB#allocateBlock}} and
> {{KeySpaceManagerProtocolClientSideTranslatorPB#openKey}} do not pass the
> ratis replication factor and replication type to the KSM server. this causes
> the allocations using ratis model to resort to standalone mode even when
> Ratis mode is specified.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12680) Ozone: SCM: Lease support for container creation
[
https://issues.apache.org/jira/browse/HDFS-12680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221110#comment-16221110
]
Nanda kumar commented on HDFS-12680:
[~anu], thanks for the review and commit. Thanks [~linyiqun] for the review.
Follow-up jiras :
* HDFS-12721 - Ozone: Ratis: Moving Ratis pipeline creation to client
* HDFS-12722 - Ozone: SCM: Lease support for pipeline creation
> Ozone: SCM: Lease support for container creation
>
>
> Key: HDFS-12680
> URL: https://issues.apache.org/jira/browse/HDFS-12680
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nanda kumar
>Assignee: Nanda kumar
> Labels: ozoneMerge
> Fix For: HDFS-7240
>
> Attachments: HDFS-12680-HDFS-7240.000.patch,
> HDFS-12680-HDFS-7240.001.patch
>
>
> This brings in lease support for container creation.
> Lease should be give for a container that is moved to {{CREATING}} state when
> {{BEGIN_CREATE}} event happens, {{LeaseException}} should be thrown if the
> container already holds a lease. Lease must be released during
> {{COMPLETE_CREATE}} event. If the lease times out container should be moved
> to {{DELETING}} state, and exception should be thrown if {{COMPLETE_CREATE}}
> event is received on that container.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Status: Patch Available (was: Open)
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12723) TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing consistently.
Rushabh S Shah created HDFS-12723:
-
Summary:
TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing
consistently.
Key: HDFS-12723
URL: https://issues.apache.org/jira/browse/HDFS-12723
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.1.0
Reporter: Rushabh S Shah
TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks is timing
out consistently on my local machine.
{noformat}
Running org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 132.405 sec <<<
FAILURE! - in org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks
testReadFileWithMissingBlocks(org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks)
Time elapsed: 132.171 sec <<< ERROR!
java.util.concurrent.TimeoutException: Timed out waiting for /foo to have all
the internalBlocks
at
org.apache.hadoop.hdfs.StripedFileTestUtil.waitBlockGroupsReported(StripedFileTestUtil.java:295)
at
org.apache.hadoop.hdfs.StripedFileTestUtil.waitBlockGroupsReported(StripedFileTestUtil.java:256)
at
org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks.readFileWithMissingBlocks(TestReadStripedFileWithMissingBlocks.java:98)
at
org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks.testReadFileWithMissingBlocks(TestReadStripedFileWithMissingBlocks.java:82)
Results :
Tests in error:
TestReadStripedFileWithMissingBlocks.testReadFileWithMissingBlocks:82->readFileWithMissingBlocks:98
» Timeout
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12521) Ozone: SCM should read all Container info into memory when booting up
[ https://issues.apache.org/jira/browse/HDFS-12521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221120#comment-16221120 ] Anu Engineer commented on HDFS-12521: - +1, I will commit this shortly. I have applied this patch and tested this locally. > Ozone: SCM should read all Container info into memory when booting up > - > > Key: HDFS-12521 > URL: https://issues.apache.org/jira/browse/HDFS-12521 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Lokesh Jain > Labels: ozoneMerge > Attachments: HDFS-12521-HDFS-7240.001.patch, > HDFS-12521-HDFS-7240.002.patch, HDFS-12521-HDFS-7240.003.patch, > HDFS-12521-HDFS-7240.004.patch, HDFS-12521-HDFS-7240.005.patch, > HDFS-12521-HDFS-7240.006.patch, HDFS-12521-HDFS-7240.007.patch, > HDFS-12521-HDFS-7240.007.patch > > > When SCM boots up it should read all containers into memory. This is a > performance optimization that allows delays on SCM side. This JIRA tracks > that issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12521) Ozone: SCM should read all Container info into memory when booting up
[ https://issues.apache.org/jira/browse/HDFS-12521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-12521: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-7240 Status: Resolved (was: Patch Available) [~linyiqun] and [~xyao] Thanks for the code reviews. [~ljain] Thanks for the contribution. I have committed this patch to the feature branch. > Ozone: SCM should read all Container info into memory when booting up > - > > Key: HDFS-12521 > URL: https://issues.apache.org/jira/browse/HDFS-12521 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Lokesh Jain > Labels: ozoneMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12521-HDFS-7240.001.patch, > HDFS-12521-HDFS-7240.002.patch, HDFS-12521-HDFS-7240.003.patch, > HDFS-12521-HDFS-7240.004.patch, HDFS-12521-HDFS-7240.005.patch, > HDFS-12521-HDFS-7240.006.patch, HDFS-12521-HDFS-7240.007.patch, > HDFS-12521-HDFS-7240.007.patch > > > When SCM boots up it should read all containers into memory. This is a > performance optimization that allows delays on SCM side. This JIRA tracks > that issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12474) Ozone: SCM: Handling container report with key count and container usage.
[ https://issues.apache.org/jira/browse/HDFS-12474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao reassigned HDFS-12474: - Assignee: Nanda kumar (was: Xiaoyu Yao) > Ozone: SCM: Handling container report with key count and container usage. > - > > Key: HDFS-12474 > URL: https://issues.apache.org/jira/browse/HDFS-12474 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Nanda kumar > Labels: ozoneMerge > > Currently, the container report only contains the # of reports sent to SCM. > We will need to provide the key count and the usage of each individual > containers to update the SCM container state maintained by > ContainerStateManager. This has a dependency on HDFS-12387. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12724) Add retry logic when data node block moves while reading
Deepak Majeti created HDFS-12724: Summary: Add retry logic when data node block moves while reading Key: HDFS-12724 URL: https://issues.apache.org/jira/browse/HDFS-12724 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Deepak Majeti Assignee: Deepak Majeti It is possible for a data node block to move due to re-balance. Libhdfs++ must try a replica block when a block re-balance happens while reading data from that block. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12720) Ozone: Ratis options are not passed from KSM Client protobuf helper correctly.
[
https://issues.apache.org/jira/browse/HDFS-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-12720:
-
Attachment: HDFS-12720-HDFS-7240.002.patch
patch v2 fixes a compilation issue in the last patch.
> Ozone: Ratis options are not passed from KSM Client protobuf helper correctly.
> --
>
> Key: HDFS-12720
> URL: https://issues.apache.org/jira/browse/HDFS-12720
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: HDFS-7240
>
> Attachments: HDFS-12720-HDFS-7240.001.patch,
> HDFS-12720-HDFS-7240.002.patch
>
>
> {{KeySpaceManagerProtocolClientSideTranslatorPB#allocateBlock}} and
> {{KeySpaceManagerProtocolClientSideTranslatorPB#openKey}} do not pass the
> ratis replication factor and replication type to the KSM server. this causes
> the allocations using ratis model to resort to standalone mode even when
> Ratis mode is specified.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12724) Add retry logic when data node block moves while reading
[ https://issues.apache.org/jira/browse/HDFS-12724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Majeti updated HDFS-12724: - Status: Patch Available (was: Open) > Add retry logic when data node block moves while reading > > > Key: HDFS-12724 > URL: https://issues.apache.org/jira/browse/HDFS-12724 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Deepak Majeti >Assignee: Deepak Majeti > > It is possible for a data node block to move due to re-balance. > Libhdfs++ must try a replica block when a block re-balance happens while > reading data from that block. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12396) Webhdfs file system should get delegation token from kms provider.
[
https://issues.apache.org/jira/browse/HDFS-12396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HDFS-12396:
--
Attachment: HDFS-12396.004.patch
All comments from v3 of patch are addressed in v4 of patch.
bq. I guess you could create a HdfsKMSUtil in the hdfs project.
Created HdfsKMSUtil class in hdfs-client package.
Since we have kms util class in hdfs-client package, I took liberty to remove
couple of kms related methods from DFSUtilClient class.
1. DFSUtilClient#createKeyProvider --> This is called only from {{FSNameSystem}}
2. DFSUtilClient#setKeyProviderUriKeyName --> Deleted this method since no-one
was calling this method.
> Webhdfs file system should get delegation token from kms provider.
> --
>
> Key: HDFS-12396
> URL: https://issues.apache.org/jira/browse/HDFS-12396
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: encryption, kms, webhdfs
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Attachments: HDFS-12396.001.patch, HDFS-12396.002.patch,
> HDFS-12396.003.patch, HDFS-12396.004.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12396) Webhdfs file system should get delegation token from kms provider.
[ https://issues.apache.org/jira/browse/HDFS-12396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HDFS-12396: -- Status: Patch Available (was: Open) > Webhdfs file system should get delegation token from kms provider. > -- > > Key: HDFS-12396 > URL: https://issues.apache.org/jira/browse/HDFS-12396 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: encryption, kms, webhdfs >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > Attachments: HDFS-12396.001.patch, HDFS-12396.002.patch, > HDFS-12396.003.patch, HDFS-12396.004.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12724) Add retry logic when data node block moves while reading
[ https://issues.apache.org/jira/browse/HDFS-12724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Majeti updated HDFS-12724: - Attachment: HDFS-12724.HDFS-8707.000.patch > Add retry logic when data node block moves while reading > > > Key: HDFS-12724 > URL: https://issues.apache.org/jira/browse/HDFS-12724 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Deepak Majeti >Assignee: Deepak Majeti > Attachments: HDFS-12724.HDFS-8707.000.patch > > > It is possible for a data node block to move due to re-balance. > Libhdfs++ must try a replica block when a block re-balance happens while > reading data from that block. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12723) TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing consistently.
[
https://issues.apache.org/jira/browse/HDFS-12723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221195#comment-16221195
]
Rushabh S Shah commented on HDFS-12723:
---
Failed in this pre-commit:
https://builds.apache.org/job/PreCommit-HDFS-Build/21784/testReport/
> TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing
> consistently.
>
>
> Key: HDFS-12723
> URL: https://issues.apache.org/jira/browse/HDFS-12723
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Rushabh S Shah
>
> TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks is timing
> out consistently on my local machine.
> {noformat}
> Running org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 132.405 sec
> <<< FAILURE! - in org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks
> testReadFileWithMissingBlocks(org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks)
> Time elapsed: 132.171 sec <<< ERROR!
> java.util.concurrent.TimeoutException: Timed out waiting for /foo to have all
> the internalBlocks
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.waitBlockGroupsReported(StripedFileTestUtil.java:295)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.waitBlockGroupsReported(StripedFileTestUtil.java:256)
> at
> org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks.readFileWithMissingBlocks(TestReadStripedFileWithMissingBlocks.java:98)
> at
> org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks.testReadFileWithMissingBlocks(TestReadStripedFileWithMissingBlocks.java:82)
> Results :
> Tests in error:
>
> TestReadStripedFileWithMissingBlocks.testReadFileWithMissingBlocks:82->readFileWithMissingBlocks:98
> » Timeout
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12667) KMSClientProvider#ValueQueue does synchronous fetch of edeks in background async thread.
[
https://issues.apache.org/jira/browse/HDFS-12667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HDFS-12667:
--
Status: Open (was: Patch Available)
> KMSClientProvider#ValueQueue does synchronous fetch of edeks in background
> async thread.
>
>
> Key: HDFS-12667
> URL: https://issues.apache.org/jira/browse/HDFS-12667
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, kms
>Affects Versions: 3.0.0-alpha4
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>
> There are couple of issues in KMSClientProvider#ValueQueue.
> 1.
> {code:title=ValueQueue.java|borderStyle=solid}
> private final LoadingCache> keyQueues;
> // Stripped rwlocks based on key name to synchronize the queue from
> // the sync'ed rw-thread and the background async refill thread.
> private final List lockArray =
> new ArrayList<>(LOCK_ARRAY_SIZE);
> {code}
> It hashes the key name into 16 buckets.
> In the code chunk below,
> {code:title=ValueQueue.java|borderStyle=solid}
> public List getAtMost(String keyName, int num) throws IOException,
> ExecutionException {
> ...
> ...
> readLock(keyName);
> E val = keyQueue.poll();
> readUnlock(keyName);
> ...
> }
> private void submitRefillTask(final String keyName,
> final Queue keyQueue) throws InterruptedException {
> ...
> ...
> writeLock(keyName); // It holds the write lock while the key is
> being asynchronously fetched. So the read requests for all the keys that
> hashes to this bucket will essentially be blocked.
> try {
> if (keyQueue.size() < threshold && !isCanceled()) {
> refiller.fillQueueForKey(name, keyQueue,
> cacheSize - keyQueue.size());
> }
> ...
> } finally {
> writeUnlock(keyName);
> }
> }
> }
> {code}
> According to above code chunk, if two keys (lets say key1 and key2) hashes to
> the same bucket (between 1 and 16), then if key1 is asynchronously being
> refetched then all the getKey for key2 will be blocked.
> 2. Due to stripped rw locks, the asynchronous behavior of refill keys is now
> synchronous to other handler threads.
> I understand that locks were added so that we don't kick off multiple
> asynchronous refilling thread for the same key.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12667) KMSClientProvider#ValueQueue does synchronous fetch of edeks in background async thread.
[
https://issues.apache.org/jira/browse/HDFS-12667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HDFS-12667:
--
Attachment: (was: HDFS-12667-001.patch)
> KMSClientProvider#ValueQueue does synchronous fetch of edeks in background
> async thread.
>
>
> Key: HDFS-12667
> URL: https://issues.apache.org/jira/browse/HDFS-12667
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, kms
>Affects Versions: 3.0.0-alpha4
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>
> There are couple of issues in KMSClientProvider#ValueQueue.
> 1.
> {code:title=ValueQueue.java|borderStyle=solid}
> private final LoadingCache> keyQueues;
> // Stripped rwlocks based on key name to synchronize the queue from
> // the sync'ed rw-thread and the background async refill thread.
> private final List lockArray =
> new ArrayList<>(LOCK_ARRAY_SIZE);
> {code}
> It hashes the key name into 16 buckets.
> In the code chunk below,
> {code:title=ValueQueue.java|borderStyle=solid}
> public List getAtMost(String keyName, int num) throws IOException,
> ExecutionException {
> ...
> ...
> readLock(keyName);
> E val = keyQueue.poll();
> readUnlock(keyName);
> ...
> }
> private void submitRefillTask(final String keyName,
> final Queue keyQueue) throws InterruptedException {
> ...
> ...
> writeLock(keyName); // It holds the write lock while the key is
> being asynchronously fetched. So the read requests for all the keys that
> hashes to this bucket will essentially be blocked.
> try {
> if (keyQueue.size() < threshold && !isCanceled()) {
> refiller.fillQueueForKey(name, keyQueue,
> cacheSize - keyQueue.size());
> }
> ...
> } finally {
> writeUnlock(keyName);
> }
> }
> }
> {code}
> According to above code chunk, if two keys (lets say key1 and key2) hashes to
> the same bucket (between 1 and 16), then if key1 is asynchronously being
> refetched then all the getKey for key2 will be blocked.
> 2. Due to stripped rw locks, the asynchronous behavior of refill keys is now
> synchronous to other handler threads.
> I understand that locks were added so that we don't kick off multiple
> asynchronous refilling thread for the same key.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12667) KMSClientProvider#ValueQueue does synchronous fetch of edeks in background async thread.
[
https://issues.apache.org/jira/browse/HDFS-12667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HDFS-12667:
--
Attachment: HDFS-12667-001.patch
re-attaching the same patch for jenkins to pickup.
> KMSClientProvider#ValueQueue does synchronous fetch of edeks in background
> async thread.
>
>
> Key: HDFS-12667
> URL: https://issues.apache.org/jira/browse/HDFS-12667
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, kms
>Affects Versions: 3.0.0-alpha4
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Attachments: HDFS-12667-001.patch
>
>
> There are couple of issues in KMSClientProvider#ValueQueue.
> 1.
> {code:title=ValueQueue.java|borderStyle=solid}
> private final LoadingCache> keyQueues;
> // Stripped rwlocks based on key name to synchronize the queue from
> // the sync'ed rw-thread and the background async refill thread.
> private final List lockArray =
> new ArrayList<>(LOCK_ARRAY_SIZE);
> {code}
> It hashes the key name into 16 buckets.
> In the code chunk below,
> {code:title=ValueQueue.java|borderStyle=solid}
> public List getAtMost(String keyName, int num) throws IOException,
> ExecutionException {
> ...
> ...
> readLock(keyName);
> E val = keyQueue.poll();
> readUnlock(keyName);
> ...
> }
> private void submitRefillTask(final String keyName,
> final Queue keyQueue) throws InterruptedException {
> ...
> ...
> writeLock(keyName); // It holds the write lock while the key is
> being asynchronously fetched. So the read requests for all the keys that
> hashes to this bucket will essentially be blocked.
> try {
> if (keyQueue.size() < threshold && !isCanceled()) {
> refiller.fillQueueForKey(name, keyQueue,
> cacheSize - keyQueue.size());
> }
> ...
> } finally {
> writeUnlock(keyName);
> }
> }
> }
> {code}
> According to above code chunk, if two keys (lets say key1 and key2) hashes to
> the same bucket (between 1 and 16), then if key1 is asynchronously being
> refetched then all the getKey for key2 will be blocked.
> 2. Due to stripped rw locks, the asynchronous behavior of refill keys is now
> synchronous to other handler threads.
> I understand that locks were added so that we don't kick off multiple
> asynchronous refilling thread for the same key.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12667) KMSClientProvider#ValueQueue does synchronous fetch of edeks in background async thread.
[
https://issues.apache.org/jira/browse/HDFS-12667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HDFS-12667:
--
Status: Patch Available (was: Open)
> KMSClientProvider#ValueQueue does synchronous fetch of edeks in background
> async thread.
>
>
> Key: HDFS-12667
> URL: https://issues.apache.org/jira/browse/HDFS-12667
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, kms
>Affects Versions: 3.0.0-alpha4
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Attachments: HDFS-12667-001.patch
>
>
> There are couple of issues in KMSClientProvider#ValueQueue.
> 1.
> {code:title=ValueQueue.java|borderStyle=solid}
> private final LoadingCache> keyQueues;
> // Stripped rwlocks based on key name to synchronize the queue from
> // the sync'ed rw-thread and the background async refill thread.
> private final List lockArray =
> new ArrayList<>(LOCK_ARRAY_SIZE);
> {code}
> It hashes the key name into 16 buckets.
> In the code chunk below,
> {code:title=ValueQueue.java|borderStyle=solid}
> public List getAtMost(String keyName, int num) throws IOException,
> ExecutionException {
> ...
> ...
> readLock(keyName);
> E val = keyQueue.poll();
> readUnlock(keyName);
> ...
> }
> private void submitRefillTask(final String keyName,
> final Queue keyQueue) throws InterruptedException {
> ...
> ...
> writeLock(keyName); // It holds the write lock while the key is
> being asynchronously fetched. So the read requests for all the keys that
> hashes to this bucket will essentially be blocked.
> try {
> if (keyQueue.size() < threshold && !isCanceled()) {
> refiller.fillQueueForKey(name, keyQueue,
> cacheSize - keyQueue.size());
> }
> ...
> } finally {
> writeUnlock(keyName);
> }
> }
> }
> {code}
> According to above code chunk, if two keys (lets say key1 and key2) hashes to
> the same bucket (between 1 and 16), then if key1 is asynchronously being
> refetched then all the getKey for key2 will be blocked.
> 2. Due to stripped rw locks, the asynchronous behavior of refill keys is now
> synchronous to other handler threads.
> I understand that locks were added so that we don't kick off multiple
> asynchronous refilling thread for the same key.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12725) BlockPlacementPolicyRackFaultTolerant still fails with racks with very few nodes
Xiao Chen created HDFS-12725: Summary: BlockPlacementPolicyRackFaultTolerant still fails with racks with very few nodes Key: HDFS-12725 URL: https://issues.apache.org/jira/browse/HDFS-12725 Project: Hadoop HDFS Issue Type: Bug Components: erasure-coding Affects Versions: 3.0.0 Reporter: Xiao Chen Assignee: Xiao Chen HDFS-12567 tries to fix the scenario where EC blocks may not be allocated in extremely rack-imbalanced cluster. The added fall-back step of the fix could be improved to do a best-effort placement. This is more likely to happen in testing than in real clusters. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12725) BlockPlacementPolicyRackFaultTolerant still fails with racks with very few nodes
[ https://issues.apache.org/jira/browse/HDFS-12725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-12725: - Attachment: HDFS-12725.01.patch Patch 1 to reproduce the error and fix. [~andrew.wang] and [~eddyxu], could you take a look? > BlockPlacementPolicyRackFaultTolerant still fails with racks with very few > nodes > > > Key: HDFS-12725 > URL: https://issues.apache.org/jira/browse/HDFS-12725 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0 >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HDFS-12725.01.patch, HDFS-12725.01.patch > > > HDFS-12567 tries to fix the scenario where EC blocks may not be allocated in > extremely rack-imbalanced cluster. > The added fall-back step of the fix could be improved to do a best-effort > placement. This is more likely to happen in testing than in real clusters. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12725) BlockPlacementPolicyRackFaultTolerant still fails with racks with very few nodes
[ https://issues.apache.org/jira/browse/HDFS-12725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-12725: - Attachment: HDFS-12725.01.patch > BlockPlacementPolicyRackFaultTolerant still fails with racks with very few > nodes > > > Key: HDFS-12725 > URL: https://issues.apache.org/jira/browse/HDFS-12725 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0 >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HDFS-12725.01.patch, HDFS-12725.01.patch > > > HDFS-12567 tries to fix the scenario where EC blocks may not be allocated in > extremely rack-imbalanced cluster. > The added fall-back step of the fix could be improved to do a best-effort > placement. This is more likely to happen in testing than in real clusters. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12725) BlockPlacementPolicyRackFaultTolerant still fails with racks with very few nodes
[
https://issues.apache.org/jira/browse/HDFS-12725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221224#comment-16221224
]
Xiao Chen edited comment on HDFS-12725 at 10/26/17 8:54 PM:
Patch 1 to reproduce the error and fix.
This keeps the {{getMaxNodesPerRack}} calculation so normal placements can
still be as even as possible. Modified unit test fail-before-pass-after.
In real clusters this come in the form of message {{Cannot allocate parity
block}} in the client and
{{org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to
place enough replicas, still in need of 3 to reach 14}} in the NN.
[~andrew.wang] and [~eddyxu], could you take a look?
was (Author: xiaochen):
Patch 1 to reproduce the error and fix. [~andrew.wang] and [~eddyxu], could you
take a look?
> BlockPlacementPolicyRackFaultTolerant still fails with racks with very few
> nodes
>
>
> Key: HDFS-12725
> URL: https://issues.apache.org/jira/browse/HDFS-12725
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-12725.01.patch, HDFS-12725.01.patch
>
>
> HDFS-12567 tries to fix the scenario where EC blocks may not be allocated in
> extremely rack-imbalanced cluster.
> The added fall-back step of the fix could be improved to do a best-effort
> placement. This is more likely to happen in testing than in real clusters.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12725) BlockPlacementPolicyRackFaultTolerant still fails with racks with very few nodes
[ https://issues.apache.org/jira/browse/HDFS-12725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-12725: - Status: Patch Available (was: Open) > BlockPlacementPolicyRackFaultTolerant still fails with racks with very few > nodes > > > Key: HDFS-12725 > URL: https://issues.apache.org/jira/browse/HDFS-12725 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0 >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HDFS-12725.01.patch > > > HDFS-12567 tries to fix the scenario where EC blocks may not be allocated in > extremely rack-imbalanced cluster. > The added fall-back step of the fix could be improved to do a best-effort > placement. This is more likely to happen in testing than in real clusters. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12725) BlockPlacementPolicyRackFaultTolerant still fails with racks with very few nodes
[ https://issues.apache.org/jira/browse/HDFS-12725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-12725: - Attachment: (was: HDFS-12725.01.patch) > BlockPlacementPolicyRackFaultTolerant still fails with racks with very few > nodes > > > Key: HDFS-12725 > URL: https://issues.apache.org/jira/browse/HDFS-12725 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0 >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HDFS-12725.01.patch > > > HDFS-12567 tries to fix the scenario where EC blocks may not be allocated in > extremely rack-imbalanced cluster. > The added fall-back step of the fix could be improved to do a best-effort > placement. This is more likely to happen in testing than in real clusters. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12711) deadly hdfs test
[
https://issues.apache.org/jira/browse/HDFS-12711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221229#comment-16221229
]
Hadoop QA commented on HDFS-12711:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
34s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 30m 49s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:blue}0{color} | {color:blue} asflicense {color} | {color:blue} 0m
14s{color} | {color:blue} ASF License check generated no output? {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestHftpFileSystem |
| Timed out junit tests | org.apache.hadoop.hdfs.TestBlockStoragePolicy |
| | org.apache.hadoop.hdfs.web.TestWebHDFS |
| | org.apache.hadoop.hdfs.TestRollingUpgradeRollback |
| | org.apache.hadoop.hdfs.web.TestWebHDFSAcl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:17213a0 |
| JIRA Issue | HDFS-12711 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12893963/HDFS-12711.branch-2.00.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux 9c99ee79afc9 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / 4678315 |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| Default Java | 1.7.0_151 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build2/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build2/3/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build2/3/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> deadly hdfs test
>
>
> Key: HDFS-12711
> URL: https://issues.apache.org/jira/browse/HDFS-12711
> Project: Hadoop HDFS
> Issue Type: Test
>Affects Versions: 2.9.0, 2.8.2
>Reporter: Allen Wittenauer
>Priority: Critical
> Attachments: HDFS-12711.branch-2.00.patch
>
>
--
This me
[jira] [Created] (HDFS-12726) BlockPlacementPolicyDefault's debugLoggingBuilder is not logged
Xiao Chen created HDFS-12726:
Summary: BlockPlacementPolicyDefault's debugLoggingBuilder is not
logged
Key: HDFS-12726
URL: https://issues.apache.org/jira/browse/HDFS-12726
Project: Hadoop HDFS
Issue Type: Bug
Components: logging
Reporter: Xiao Chen
Assignee: Xiao Chen
During debugging HDFS-12725, {{BlockPlacementPolicyDefault's}} class'
{{debugLoggingBuilder}} does a lot of {{get}} and {{append}}, but never
{{toString}} and {{LOG.debug}}'ed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12100) Ozone: KSM: Allocate key should honour volume quota if quota is set on the volume
[ https://issues.apache.org/jira/browse/HDFS-12100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221234#comment-16221234 ] Xiaoyu Yao commented on HDFS-12100: --- [~ljain], thanks for working on this. Can you rebase the patch? > Ozone: KSM: Allocate key should honour volume quota if quota is set on the > volume > - > > Key: HDFS-12100 > URL: https://issues.apache.org/jira/browse/HDFS-12100 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Lokesh Jain > Labels: OzonePostMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12100-HDFS-7240.001.patch, > HDFS-12100-HDFS-7240.002.patch, HDFS-12100-HDFS-7240.003.patch > > > KeyManagerImpl#allocateKey currently does not check the volume quota before > allocating a key, this can cause the volume quota overrun. > Volume quota needs to be check before allocating the key in the SCM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12727) TestPread timing out on branch-2.8
Rushabh S Shah created HDFS-12727:
-
Summary: TestPread timing out on branch-2.8
Key: HDFS-12727
URL: https://issues.apache.org/jira/browse/HDFS-12727
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Rushabh S Shah
TestPread timing out on branch-2.8 and not on trunk.
Few lines before hanging.
{noformat}
2017-10-26 20:21:07,938 WARN impl.BlockReaderFactory
(BlockReaderFactory.java:getRemoteBlockReaderFromTcp(758)) - I/O error
constructing remote block reader.
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:534)
at
org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2955)
at
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:815)
at
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:740)
at
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:385)
at
org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:708)
at
org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1230)
at
org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1198)
at
org.apache.hadoop.hdfs.DFSInputStream.access$000(DFSInputStream.java:97)
at
org.apache.hadoop.hdfs.DFSInputStream$2.call(DFSInputStream.java:1182)
at
org.apache.hadoop.hdfs.DFSInputStream$2.call(DFSInputStream.java:1174)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-10-26 20:21:07,938 WARN hdfs.DFSClient
(DFSInputStream.java:actualGetFromOneDataNode(1270)) - Connection failure:
Failed to connect to /127.0.0.1:42357 for file /preadtest.dat for block
BP-287215640-172.17.0.18-1509049266453:blk_1073741825_1001:java.net.ConnectException:
Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:534)
at
org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2955)
at
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:815)
at
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:740)
at
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:385)
at
org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:708)
at
org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1230)
at
org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1198)
at
org.apache.hadoop.hdfs.DFSInputStream.access$000(DFSInputStream.java:97)
at
org.apache.hadoop.hdfs.DFSInputStream$2.call(DFSInputStream.java:1182)
at
org.apache.hadoop.hdfs.DFSInputStream$2.call(DFSInputStream.java:1174)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-10-26 20:21:07,939 WARN hdfs.DFSClient
(DFSInputStream.java:getBestNodeDNAddrPair(1112)) - No live nodes contain block
BP-287215640-172.17.0.18-1509049266453:blk_1073741825_1001 after checking nodes
=
[DatanodeInfoWithStorage[127.0.0.1:36669,DS-3ce766bc-dad8-4022-b0d8-396a669ee4b8,DISK],
DatanodeInfoWithStorage[127.0.0.1:36005,DS-bd7b59f4-a7de-4524-877e-c0f9a10ce5d5,DISK],
DatanodeInfoWithStorage[127.0.0.1:42357,DS-a42e4a89-3985-4dc7-ad6a-c0bcf078ccae,DISK]],
ignoredNodes =
[DatanodeInfoWithStorage[127.0.0.1:36005,DS-bd7b59f4-a7de-4524-877e-c0f9a10ce5d5,
[jira] [Updated] (HDFS-12727) TestPread timing out on branch-2.8
[
https://issues.apache.org/jira/browse/HDFS-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HDFS-12727:
--
Description:
TestPread timing out on branch-2.8 and not on trunk.
{noformat}
2017-10-24 19:47:37,377 WARN impl.BlockReaderFactory
(BlockReaderFactory.java:getRemoteBlockReaderFromTcp(758)) - I/O error
constructing remote block reader.
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:534)
at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2955)
at
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:815)
at
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:740)
at
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:385)
at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:708)
at
org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1230)
at
org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1198)
at
org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1158)
at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1535)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1501)
at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
at org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:120)
at org.apache.hadoop.hdfs.TestPread.datanodeRestartTest(TestPread.java:245)
at org.apache.hadoop.hdfs.TestPread.dfsPreadTest(TestPread.java:478)
at org.apache.hadoop.hdfs.TestPread.testPreadDFSNoChecksum(TestPread.java:280)
{noformat}
Few lines in the log before hanging.
{noformat}
2017-10-26 20:21:07,938 WARN impl.BlockReaderFactory
(BlockReaderFactory.java:getRemoteBlockReaderFromTcp(758)) - I/O error
constructing remote block reader.
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:534)
at
org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2955)
at
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:815)
at
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:740)
at
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:385)
at
org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:708)
at
org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1230)
at
org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1198)
at
org.apache.hadoop.hdfs.DFSInputStream.access$000(DFSInputStream.java:97)
at
org.apache.hadoop.hdfs.DFSInputStream$2.call(DFSInputStream.java:1182)
at
org.apache.hadoop.hdfs.DFSInputStream$2.call(DFSInputStream.java:1174)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-10-26 20:21:07,938 WARN hdfs.DFSClient
(DFSInputStream.java:actualGetFromOneDataNode(1270)) - Connection failure:
Failed to connect to /127.0.0.1:42357 for file /preadtest.dat for block
BP-287215640-172.17.0.18-1509049266453:blk_1073741825_1001:java.net.ConnectException:
Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:534)
at
org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2955)
at
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:815)
at
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:740)
at
or
[jira] [Updated] (HDFS-12720) Ozone: Ratis options are not passed from KSM Client protobuf helper correctly.
[
https://issues.apache.org/jira/browse/HDFS-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HDFS-12720:
Labels: ozoneMerge (was: )
> Ozone: Ratis options are not passed from KSM Client protobuf helper correctly.
> --
>
> Key: HDFS-12720
> URL: https://issues.apache.org/jira/browse/HDFS-12720
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Labels: ozoneMerge
> Fix For: HDFS-7240
>
> Attachments: HDFS-12720-HDFS-7240.001.patch,
> HDFS-12720-HDFS-7240.002.patch
>
>
> {{KeySpaceManagerProtocolClientSideTranslatorPB#allocateBlock}} and
> {{KeySpaceManagerProtocolClientSideTranslatorPB#openKey}} do not pass the
> ratis replication factor and replication type to the KSM server. this causes
> the allocations using ratis model to resort to standalone mode even when
> Ratis mode is specified.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12607) [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED
[
https://issues.apache.org/jira/browse/HDFS-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12607:
--
Status: Open (was: Patch Available)
> [READ] Even one dead datanode with PROVIDED storage results in
> ProvidedStorageInfo being marked as FAILED
> -
>
> Key: HDFS-12607
> URL: https://issues.apache.org/jira/browse/HDFS-12607
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12607-HDFS-9806.001.patch,
> HDFS-12607-HDFS-9806.002.patch, HDFS-12607.repro.patch
>
>
> When a DN configured with PROVIDED storage is marked as dead by the NN, the
> state of {{providedStorageInfo}} in {{ProvidedStorageMap}} is set to FAILED,
> and never becomes NORMAL. The state should change to FAILED only if all
> datanodes with PROVIDED storage are dead, and should be restored back to
> NORMAL when a Datanode with NORMAL DatanodeStorage reports in.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12607) [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED
[
https://issues.apache.org/jira/browse/HDFS-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12607:
--
Status: Patch Available (was: Open)
> [READ] Even one dead datanode with PROVIDED storage results in
> ProvidedStorageInfo being marked as FAILED
> -
>
> Key: HDFS-12607
> URL: https://issues.apache.org/jira/browse/HDFS-12607
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12607-HDFS-9806.001.patch,
> HDFS-12607-HDFS-9806.002.patch, HDFS-12607.repro.patch
>
>
> When a DN configured with PROVIDED storage is marked as dead by the NN, the
> state of {{providedStorageInfo}} in {{ProvidedStorageMap}} is set to FAILED,
> and never becomes NORMAL. The state should change to FAILED only if all
> datanodes with PROVIDED storage are dead, and should be restored back to
> NORMAL when a Datanode with NORMAL DatanodeStorage reports in.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
