[jira] [Commented] (HDFS-7770) Need document for storage type label of data node storage locations under dfs.data.dir
[
https://issues.apache.org/jira/browse/HDFS-7770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328798#comment-14328798
]
Hadoop QA commented on HDFS-7770:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12699828/HDFS-7770.00.patch
against trunk revision c0d9b93.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster
org.apache.hadoop.hdfs.TestLeaseRecovery2
The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.fs.TestSymlinkHdfsFileContext
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9627//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9627//console
This message is automatically generated.
Need document for storage type label of data node storage locations under
dfs.data.dir
--
Key: HDFS-7770
URL: https://issues.apache.org/jira/browse/HDFS-7770
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Attachments: HDFS-7770.00.patch
HDFS-2832 enables support for heterogeneous storages in HDFS, which allows DN
as a collection of storages with different types. However, I can't find
document on how to label different storage types from the following two
documents. I found the information from the design spec. It will be good we
document this for admins and users to use the related Archival storage and
storage policy features.
http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html
http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
This JIRA is opened to add document for the new storage type labels.
1. Add an example under ArchivalStorage.html#Configuration section:
{code}
property
namedfs.data.dir/name
value[DISK]file:///hddata/dn/disk0,
[SSD]file:///hddata/dn/ssd0,[ARCHIVE]file:///hddata/dn/archive0/value
/property
{code}
2. Add a short description of [DISK/SSD/ARCHIVE/RAM_DISK] options in
hdfs-default.xml#dfs.data.dir and document DISK as storage type if no storage
type is labeled in the data node storage location configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7700) Update document for quota by storage type
[
https://issues.apache.org/jira/browse/HDFS-7700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328667#comment-14328667
]
Hadoop QA commented on HDFS-7700:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12699813/HDFS-7700.00.patch
against trunk revision c0d9b93.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9626//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9626//console
This message is automatically generated.
Update document for quota by storage type
-
Key: HDFS-7700
URL: https://issues.apache.org/jira/browse/HDFS-7700
Project: Hadoop HDFS
Issue Type: Sub-task
Components: documentation
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Attachments: HDFS-7700.00.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HDFS-7740) Test truncate with DataNodes restarting
[
https://issues.apache.org/jira/browse/HDFS-7740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328679#comment-14328679
]
Yi Liu edited comment on HDFS-7740 at 2/20/15 8:35 AM:
---
Thanks [~shv] for your comments. I will address them later, currently I'am on
holiday for Chinese traditional new year, so there is some delay for the update.
{quote}
I don't think your new tests will work with the ones existing in
TestFileTruncate, because you are restarting the whole cluster
{quote}
Oh, not noticed these before :) I thought if I use a different Mini DFS Cluster
with different ports, then they will not affect each other.
{quote}
I would propose to put your new cases into a new file
I think this is possible as you do not need to reformat the cluster after each
test
{quote}
Good idea, I will move them to a new class. Yes, we don't need to reformat the
cluster.
{quote}
It would be good to start the cluster once in @BeforeClass, and make sure all
DNs are up after each test case.
{quote}
To make sure all DNs are up after each test case, I think we are referring to
use @Before instead of @BeforeClass
{quote}
With DNs restarting one way to accelerate test running time is to shorten
heartbeats, as we did in TestFileTruncate.
{quote}
We already use the short hearbeats for the new test.
was (Author: hitliuyi):
Thanks [~shv] for your comments. I will address them later, currently I'am on
holiday for Chinese traditional new year, so there is some delay for the update.
{quote}
I don't think your new tests will work with the ones existing in
TestFileTruncate, because you are restarting the whole cluster
{quote}
Oh, not noticed these before :) I thought if I use a different Mini DFS Cluster
with different ports, then they will not affect each other.
{quote}
I would propose to put your new cases into a new file
It would be good to start the cluster once in @BeforeClass
I think this is possible as you do not need to reformat the cluster after each
test
{quote}
Good idea, I will move them to a new class, and then we can use @BeforeClass.
Yes, we don't need to reformat the cluster.
{quote}
With DNs restarting one way to accelerate test running time is to shorten
heartbeats, as we did in TestFileTruncate.
{quote}
We already use the short hearbeats for the new test.
Test truncate with DataNodes restarting
---
Key: HDFS-7740
URL: https://issues.apache.org/jira/browse/HDFS-7740
Project: Hadoop HDFS
Issue Type: Sub-task
Components: test
Affects Versions: 2.7.0
Reporter: Konstantin Shvachko
Assignee: Yi Liu
Fix For: 2.7.0
Attachments: HDFS-7740.001.patch
Add a test case, which ensures replica consistency when DNs are failing and
restarting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HDFS-7740) Test truncate with DataNodes restarting
[
https://issues.apache.org/jira/browse/HDFS-7740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328679#comment-14328679
]
Yi Liu edited comment on HDFS-7740 at 2/20/15 8:47 AM:
---
Thanks [~shv] for your comments. I will address them later, currently I'am on
holiday for Chinese traditional new year, so there is some delay for the update.
{quote}
I don't think your new tests will work with the ones existing in
TestFileTruncate, because you are restarting the whole cluster
{quote}
Oh, not noticed these before :) I thought if I use a different Mini DFS Cluster
with different ports, then they will not affect each other.
{quote}
I would propose to put your new cases into a new file
It would be good to start the cluster once in @BeforeClass, and make sure all
DNs are up after each test case.
I think this is possible as you do not need to reformat the cluster after each
test
{quote}
Good idea, I will move them to a new class. Yes, we don't need to reformat the
cluster.
{quote}
With DNs restarting one way to accelerate test running time is to shorten
heartbeats, as we did in TestFileTruncate.
{quote}
We already use the short hearbeats for the new test.
was (Author: hitliuyi):
Thanks [~shv] for your comments. I will address them later, currently I'am on
holiday for Chinese traditional new year, so there is some delay for the update.
{quote}
I don't think your new tests will work with the ones existing in
TestFileTruncate, because you are restarting the whole cluster
{quote}
Oh, not noticed these before :) I thought if I use a different Mini DFS Cluster
with different ports, then they will not affect each other.
{quote}
I would propose to put your new cases into a new file
I think this is possible as you do not need to reformat the cluster after each
test
{quote}
Good idea, I will move them to a new class. Yes, we don't need to reformat the
cluster.
{quote}
It would be good to start the cluster once in @BeforeClass, and make sure all
DNs are up after each test case.
{quote}
To make sure all DNs are up after each test case, I think we are referring to
use @Before instead of @BeforeClass
{quote}
With DNs restarting one way to accelerate test running time is to shorten
heartbeats, as we did in TestFileTruncate.
{quote}
We already use the short hearbeats for the new test.
Test truncate with DataNodes restarting
---
Key: HDFS-7740
URL: https://issues.apache.org/jira/browse/HDFS-7740
Project: Hadoop HDFS
Issue Type: Sub-task
Components: test
Affects Versions: 2.7.0
Reporter: Konstantin Shvachko
Assignee: Yi Liu
Fix For: 2.7.0
Attachments: HDFS-7740.001.patch
Add a test case, which ensures replica consistency when DNs are failing and
restarting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7740) Test truncate with DataNodes restarting
[
https://issues.apache.org/jira/browse/HDFS-7740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu updated HDFS-7740:
-
Attachment: HDFS-7740.002.patch
[~shv], since we use 3 datanodes in {{TestFileTruncate}}, how about we still
keep these tests in the same file, and we don't need to use separate
MiniDFSCluster for each test case. Of course, we make sure all the DNs are up
after each test case.
Update the patch, please take a look.
Test truncate with DataNodes restarting
---
Key: HDFS-7740
URL: https://issues.apache.org/jira/browse/HDFS-7740
Project: Hadoop HDFS
Issue Type: Sub-task
Components: test
Affects Versions: 2.7.0
Reporter: Konstantin Shvachko
Assignee: Yi Liu
Fix For: 2.7.0
Attachments: HDFS-7740.001.patch, HDFS-7740.002.patch
Add a test case, which ensures replica consistency when DNs are failing and
restarting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HDFS-7740) Test truncate with DataNodes restarting
[
https://issues.apache.org/jira/browse/HDFS-7740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328679#comment-14328679
]
Yi Liu edited comment on HDFS-7740 at 2/20/15 9:14 AM:
---
Thanks [~shv] for your comments. I will address them later, currently I'am on
holiday for Chinese traditional new year, so there is some delay for the update.
{quote}
I don't think your new tests will work with the ones existing in
TestFileTruncate, because you are restarting the whole cluster
{quote}
Oh, not noticed these before :) I thought if I use a different Mini DFS Cluster
with different ports, then they will not affect each other.
I run these new added test cases separately and they worked fine, so not notice
they affect other tests.
{quote}
I would propose to put your new cases into a new file
It would be good to start the cluster once in @BeforeClass, and make sure all
DNs are up after each test case.
I think this is possible as you do not need to reformat the cluster after each
test
{quote}
Good idea, I will move them to a new class. Yes, we don't need to reformat the
cluster.
{quote}
With DNs restarting one way to accelerate test running time is to shorten
heartbeats, as we did in TestFileTruncate.
{quote}
We already use the short hearbeats for the new test.
was (Author: hitliuyi):
Thanks [~shv] for your comments. I will address them later, currently I'am on
holiday for Chinese traditional new year, so there is some delay for the update.
{quote}
I don't think your new tests will work with the ones existing in
TestFileTruncate, because you are restarting the whole cluster
{quote}
Oh, not noticed these before :) I thought if I use a different Mini DFS Cluster
with different ports, then they will not affect each other.
{quote}
I would propose to put your new cases into a new file
It would be good to start the cluster once in @BeforeClass, and make sure all
DNs are up after each test case.
I think this is possible as you do not need to reformat the cluster after each
test
{quote}
Good idea, I will move them to a new class. Yes, we don't need to reformat the
cluster.
{quote}
With DNs restarting one way to accelerate test running time is to shorten
heartbeats, as we did in TestFileTruncate.
{quote}
We already use the short hearbeats for the new test.
Test truncate with DataNodes restarting
---
Key: HDFS-7740
URL: https://issues.apache.org/jira/browse/HDFS-7740
Project: Hadoop HDFS
Issue Type: Sub-task
Components: test
Affects Versions: 2.7.0
Reporter: Konstantin Shvachko
Assignee: Yi Liu
Fix For: 2.7.0
Attachments: HDFS-7740.001.patch
Add a test case, which ensures replica consistency when DNs are failing and
restarting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7740) Test truncate with DataNodes restarting
[
https://issues.apache.org/jira/browse/HDFS-7740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328679#comment-14328679
]
Yi Liu commented on HDFS-7740:
--
Thanks [~shv] for your comments. I will address them later, currently I'am on
holiday for Chinese traditional new year, so there is some delay for the update.
{quote}
I don't think your new tests will work with the ones existing in
TestFileTruncate, because you are restarting the whole cluster
{quote}
Oh, not noticed these before :) I thought if I use a different Mini DFS Cluster
with different ports, then they will not affect each other.
{quote}
I would propose to put your new cases into a new file
It would be good to start the cluster once in @BeforeClass
I think this is possible as you do not need to reformat the cluster after each
test
{quote}
Good idea, I will move them to a new class, and then we can use @BeforeClass.
Yes, we don't need to reformat the cluster.
{quote}
With DNs restarting one way to accelerate test running time is to shorten
heartbeats, as we did in TestFileTruncate.
{quote}
We already use the short hearbeats for the new test.
Test truncate with DataNodes restarting
---
Key: HDFS-7740
URL: https://issues.apache.org/jira/browse/HDFS-7740
Project: Hadoop HDFS
Issue Type: Sub-task
Components: test
Affects Versions: 2.7.0
Reporter: Konstantin Shvachko
Assignee: Yi Liu
Fix For: 2.7.0
Attachments: HDFS-7740.001.patch
Add a test case, which ensures replica consistency when DNs are failing and
restarting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7815) Loop on 'blocks does not belong to any file'
Frode Halvorsen created HDFS-7815: - Summary: Loop on 'blocks does not belong to any file' Key: HDFS-7815 URL: https://issues.apache.org/jira/browse/HDFS-7815 Project: Hadoop HDFS Issue Type: Bug Components: datanode, namenode Affects Versions: 2.6.0 Environment: small cluster on RetHat. 2 namenodes (HA), 6 datanodes with 19TB disk for hdfs. Reporter: Frode Halvorsen I am currently experincing a looping situation; The namenode uses appx 1:50 (min:sec) to log a massive amount of lines stating that some blocks don't belong to any file. During this time, it's unresponsive to any requests from datanodes, and if the zoo-keper had been running, it would have taken the name-node down (ssh-fencing : kill). When it has finished the 'round', it starts to do some normal work, and among other things, telling the datanode to delete the blocks. But before the datanode has gotten around to delete the blocks, and is about to report back to the namenode, the namenode has stared on the next round of reporing the same blocks that don't belong to anly file. Thus, the datanode gets a timout when reporing block-updates for the deleted blocks, And this, of course repeats itself over and over again... There is actually two issues , I think,; 1- the namenode gets totally unresponsive when reporing the blocks (could this be a debug-line instead of a INFO-line) 2 - the namenode seems to 'forget' that it has already reported those blocks just 2-3 minutes ago... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7740) Test truncate with DataNodes restarting
[
https://issues.apache.org/jira/browse/HDFS-7740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328870#comment-14328870
]
Hadoop QA commented on HDFS-7740:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12699840/HDFS-7740.002.patch
against trunk revision c0d9b93.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9628//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9628//console
This message is automatically generated.
Test truncate with DataNodes restarting
---
Key: HDFS-7740
URL: https://issues.apache.org/jira/browse/HDFS-7740
Project: Hadoop HDFS
Issue Type: Sub-task
Components: test
Affects Versions: 2.7.0
Reporter: Konstantin Shvachko
Assignee: Yi Liu
Fix For: 2.7.0
Attachments: HDFS-7740.001.patch, HDFS-7740.002.patch
Add a test case, which ensures replica consistency when DNs are failing and
restarting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7752) Improve description for dfs.namenode.num.extra.edits.retained and dfs.namenode.num.checkpoints.retained properties on hdfs-default.xml
[ https://issues.apache.org/jira/browse/HDFS-7752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HDFS-7752: -- Target Version/s: 2.7.0 Hadoop Flags: Reviewed Improve description for dfs.namenode.num.extra.edits.retained and dfs.namenode.num.checkpoints.retained properties on hdfs-default.xml -- Key: HDFS-7752 URL: https://issues.apache.org/jira/browse/HDFS-7752 Project: Hadoop HDFS Issue Type: Improvement Components: documentation Affects Versions: 2.6.0 Reporter: Wellington Chevreuil Assignee: Wellington Chevreuil Priority: Minor Attachments: HDFS-7752.patch, HDFS-7752.patch Current description for dfs.namenode.num.extra.edits.retained and dfs.namenode.num.checkpoints.retained properties on hdfs-default.xml is not clear on how much and which files will be kept on namenodes meta-data directory. For dfs.namenode.num.checkpoints.retained, it's not clear that it applies to the number of fsimage_* files. For dfs.namenode.num.extra.edits.retained, it's not clear the value set indirectly applies to edits_* files, and how the configured value translates into the number of edit files to be kept. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6133) Make Balancer support exclude specified path
[
https://issues.apache.org/jira/browse/HDFS-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328914#comment-14328914
]
Yongjun Zhang commented on HDFS-6133:
-
HI [~cmccabe],
My understanding is that HDFS-4420's approach is a passive pinning: it tells
balancer not to move the blocks belonging to a specified file path; on the
other hand, HDFS-6133 proactively pin the blocks at write time by specifying
favored nodes. I think if we do passive pinning after the the blocks of a
file have been moved by balancer, then it may not be preferred behavior. Hi
[~zhaoyunjiong], would you please share your thoughts when you started with
this approach when HDFS-4420's patch was already there?
Hi [~szetszwo] and [~zhaoyunjiong], about HDFS-6133, I was wondering whether
there is any compatibility issue when we do rolling upgrade? There are API
changes at {{DataTansferProtocol#writeBlock}}, what would happen if DNs with
HDFS-6133 (trying to pin) talks to DNs without the fix? Would you please
comment?
Thanks.
Make Balancer support exclude specified path
Key: HDFS-6133
URL: https://issues.apache.org/jira/browse/HDFS-6133
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer mover, datanode
Reporter: zhaoyunjiong
Assignee: zhaoyunjiong
Fix For: 2.7.0
Attachments: HDFS-6133-1.patch, HDFS-6133-10.patch,
HDFS-6133-11.patch, HDFS-6133-2.patch, HDFS-6133-3.patch, HDFS-6133-4.patch,
HDFS-6133-5.patch, HDFS-6133-6.patch, HDFS-6133-7.patch, HDFS-6133-8.patch,
HDFS-6133-9.patch, HDFS-6133.patch
Currently, run Balancer will destroying Regionserver's data locality.
If getBlocks could exclude blocks belongs to files which have specific path
prefix, like /hbase, then we can run Balancer without destroying
Regionserver's data locality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HDFS-7308) DFSClient write packet size may 64kB
[
https://issues.apache.org/jira/browse/HDFS-7308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze reassigned HDFS-7308:
-
Assignee: Takuya Fukudome (was: Tsz Wo Nicholas Sze)
Sure, assigned to you. Thanks for working on this.
DFSClient write packet size may 64kB
--
Key: HDFS-7308
URL: https://issues.apache.org/jira/browse/HDFS-7308
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Reporter: Tsz Wo Nicholas Sze
Assignee: Takuya Fukudome
Priority: Minor
In DFSOutputStream.computePacketChunkSize(..),
{code}
private void computePacketChunkSize(int psize, int csize) {
final int chunkSize = csize + getChecksumSize();
chunksPerPacket = Math.max(psize/chunkSize, 1);
packetSize = chunkSize*chunksPerPacket;
if (DFSClient.LOG.isDebugEnabled()) {
...
}
}
{code}
We have the following
|| variables || usual values ||
| psize | dfsClient.getConf().writePacketSize = 64kB |
| csize | bytesPerChecksum = 512B |
| getChecksumSize(), i.e. CRC size | 32B |
| chunkSize = csize + getChecksumSize() | 544B (not a power of two) |
| psize/chunkSize | 120.47 |
| chunksPerPacket = max(psize/chunkSize, 1) | 120 |
| packetSize = chunkSize*chunksPerPacket (not including header) | 65280B |
| PacketHeader.PKT_MAX_HEADER_LEN | 33B |
| actual packet size | 65280 + 33 = *65313* 65536 = 64k |
It is fortunate that the usual packet size = 65313 64k although the
calculation above does not guarantee it always happens (e.g. if
PKT_MAX_HEADER_LEN=257, then actual packet size=65537 64k.) We should fix
the computation in order to guarantee actual packet size 64k.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6133) Make Balancer support exclude specified path
[ https://issues.apache.org/jira/browse/HDFS-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328931#comment-14328931 ] Tsz Wo Nicholas Sze commented on HDFS-6133: --- ... I was wondering whether there is any compatibility issue when we do rolling upgrade? ... For rolling upgrade, the block pinning feature should be first disabled, then upgrade the cluster and then enable the feature. What do you think? Make Balancer support exclude specified path Key: HDFS-6133 URL: https://issues.apache.org/jira/browse/HDFS-6133 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover, datanode Reporter: zhaoyunjiong Assignee: zhaoyunjiong Fix For: 2.7.0 Attachments: HDFS-6133-1.patch, HDFS-6133-10.patch, HDFS-6133-11.patch, HDFS-6133-2.patch, HDFS-6133-3.patch, HDFS-6133-4.patch, HDFS-6133-5.patch, HDFS-6133-6.patch, HDFS-6133-7.patch, HDFS-6133-8.patch, HDFS-6133-9.patch, HDFS-6133.patch Currently, run Balancer will destroying Regionserver's data locality. If getBlocks could exclude blocks belongs to files which have specific path prefix, like /hbase, then we can run Balancer without destroying Regionserver's data locality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7752) Improve description for dfs.namenode.num.extra.edits.retained and dfs.namenode.num.checkpoints.retained properties on hdfs-default.xml
[ https://issues.apache.org/jira/browse/HDFS-7752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HDFS-7752. --- Resolution: Fixed Fix Version/s: 2.7.0 Target Version/s: (was: 2.7.0) Thanks Wellington! I've committed this to branch-2 and trunk. Improve description for dfs.namenode.num.extra.edits.retained and dfs.namenode.num.checkpoints.retained properties on hdfs-default.xml -- Key: HDFS-7752 URL: https://issues.apache.org/jira/browse/HDFS-7752 Project: Hadoop HDFS Issue Type: Improvement Components: documentation Affects Versions: 2.6.0 Reporter: Wellington Chevreuil Assignee: Wellington Chevreuil Priority: Minor Fix For: 2.7.0 Attachments: HDFS-7752.patch, HDFS-7752.patch Current description for dfs.namenode.num.extra.edits.retained and dfs.namenode.num.checkpoints.retained properties on hdfs-default.xml is not clear on how much and which files will be kept on namenodes meta-data directory. For dfs.namenode.num.checkpoints.retained, it's not clear that it applies to the number of fsimage_* files. For dfs.namenode.num.extra.edits.retained, it's not clear the value set indirectly applies to edits_* files, and how the configured value translates into the number of edit files to be kept. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6133) Make Balancer support exclude specified path
[ https://issues.apache.org/jira/browse/HDFS-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328935#comment-14328935 ] Yongjun Zhang commented on HDFS-6133: - Hi Nicholas, Thanks a lot for your quick reply! I think that means we need to change the rolling upgrade steps, which still means a soft incompatible. Should we mark this jira as incompatible? Make Balancer support exclude specified path Key: HDFS-6133 URL: https://issues.apache.org/jira/browse/HDFS-6133 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover, datanode Reporter: zhaoyunjiong Assignee: zhaoyunjiong Fix For: 2.7.0 Attachments: HDFS-6133-1.patch, HDFS-6133-10.patch, HDFS-6133-11.patch, HDFS-6133-2.patch, HDFS-6133-3.patch, HDFS-6133-4.patch, HDFS-6133-5.patch, HDFS-6133-6.patch, HDFS-6133-7.patch, HDFS-6133-8.patch, HDFS-6133-9.patch, HDFS-6133.patch Currently, run Balancer will destroying Regionserver's data locality. If getBlocks could exclude blocks belongs to files which have specific path prefix, like /hbase, then we can run Balancer without destroying Regionserver's data locality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7752) Improve description for dfs.namenode.num.extra.edits.retained and dfs.namenode.num.checkpoints.retained properties on hdfs-default.xml
[ https://issues.apache.org/jira/browse/HDFS-7752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328939#comment-14328939 ] Hudson commented on HDFS-7752: -- FAILURE: Integrated in Hadoop-trunk-Commit #7160 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7160/]) HDFS-7752. Improve description for dfs.namenode.num.extra.edits.retained and dfs.namenode.num.checkpoints.retained properties on hdfs-default.xml. Contributed by Wellington Chevreuil. (harsh: rev b9a17909ba39898120a096cb6ae90104640690db) * hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Improve description for dfs.namenode.num.extra.edits.retained and dfs.namenode.num.checkpoints.retained properties on hdfs-default.xml -- Key: HDFS-7752 URL: https://issues.apache.org/jira/browse/HDFS-7752 Project: Hadoop HDFS Issue Type: Improvement Components: documentation Affects Versions: 2.6.0 Reporter: Wellington Chevreuil Assignee: Wellington Chevreuil Priority: Minor Fix For: 2.7.0 Attachments: HDFS-7752.patch, HDFS-7752.patch Current description for dfs.namenode.num.extra.edits.retained and dfs.namenode.num.checkpoints.retained properties on hdfs-default.xml is not clear on how much and which files will be kept on namenodes meta-data directory. For dfs.namenode.num.checkpoints.retained, it's not clear that it applies to the number of fsimage_* files. For dfs.namenode.num.extra.edits.retained, it's not clear the value set indirectly applies to edits_* files, and how the configured value translates into the number of edit files to be kept. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7788) Post-2.6 namenode may not start up with an image containing inodes created with an old release.
[
https://issues.apache.org/jira/browse/HDFS-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329010#comment-14329010
]
Kihwal Lee commented on HDFS-7788:
--
Although the precommit posted a +1, the test result is missing. Judging from
the run-time, it ran many tests. So I ran hdfs tests locally overnight. Two
test cases failed.
{panel}
TestDFSHAAdminMiniCluster.testFencer:163 expected:0 but was:-1
TestDFSUpgradeFromImage.testUpgradeFromRel1BBWImage:619-upgradeAndVerify:597-verifyFileSystem:225-verifyDir:210-dfsOpenFileWithRetries:174
» IO
{panel}
I just reran the tests and they are passing.
+1 The change looks good.
Post-2.6 namenode may not start up with an image containing inodes created
with an old release.
---
Key: HDFS-7788
URL: https://issues.apache.org/jira/browse/HDFS-7788
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Kihwal Lee
Assignee: Rushabh S Shah
Priority: Blocker
Attachments: HDFS-7788-binary.patch, rushabh.patch
Before HDFS-4305, which was fixed in 2.1.0-beta, clients could specify
arbitrarily small preferred block size for a file including 0. This was
normally done by faulty clients or failed creates, but it was possible.
Until 2.5, reading a fsimage containing inodes with 0 byte preferred block
size was allowed. So if a fsimage contained such an inode, the namenode would
come up fine. In 2.6, the preferred block size is required be 0. Because
of this change, the image that worked with 2.5 may not work with 2.6.
If a cluster ran a version of hadoop earlier than 2.1.0-beta before, it is
under this risk even if it worked fine with 2.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7814) Fix usage string of storageType parameter for dfsadmin -setSpaceQuota/clrSpaceQuota
[ https://issues.apache.org/jira/browse/HDFS-7814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-7814: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) +1 for the patch. The test failures are unrelated and tracked elsewhere. I committed this to trunk and branch-2. Thank you for fixing this, Xiaoyu. Fix usage string of storageType parameter for dfsadmin -setSpaceQuota/clrSpaceQuota - Key: HDFS-7814 URL: https://issues.apache.org/jira/browse/HDFS-7814 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Priority: Minor Fix For: 2.7.0 Attachments: HDFS-7814.00.patch This was found when I'm documenting the quota by storage type feature. The current usage string to set/clear quota by storage type which put the -storageType prameter after dirnames is incorrect. hdfs dfsadmin -setSpaceQuota/clrSpaceQuota quota dirname...dirname -storageType storagetype. The correct one should be: hdfs dfsadmin -setSpaceQuota/clrSpaceQuota quota [-storageType storagetype] dirname...dirname I will post the fix shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7816) Unable to open webhdfs paths with +
[ https://issues.apache.org/jira/browse/HDFS-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-7816: - Attachment: HDFS-7816.patch The previous one was a broken patch. Attaching the correct one. Unable to open webhdfs paths with + - Key: HDFS-7816 URL: https://issues.apache.org/jira/browse/HDFS-7816 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.7.0 Reporter: Jason Lowe Assignee: Kihwal Lee Priority: Blocker Attachments: HDFS-7816.patch, HDFS-7816.patch webhdfs requests to open files with % characters in the filename fail because the filename is not being decoded properly. For example: $ hadoop fs -cat 'webhdfs://nn/user/somebody/abc%def' cat: File does not exist: /user/somebody/abc%25def -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HDFS-7816) Unable to open webhdfs paths with +
[ https://issues.apache.org/jira/browse/HDFS-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-7816: Assignee: Kihwal Lee Unable to open webhdfs paths with + - Key: HDFS-7816 URL: https://issues.apache.org/jira/browse/HDFS-7816 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.7.0 Reporter: Jason Lowe Assignee: Kihwal Lee Priority: Blocker webhdfs requests to open files with % characters in the filename fail because the filename is not being decoded properly. For example: $ hadoop fs -cat 'webhdfs://nn/user/somebody/abc%def' cat: File does not exist: /user/somebody/abc%25def -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7816) Unable to open webhdfs paths with +
[ https://issues.apache.org/jira/browse/HDFS-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-7816: - Status: Patch Available (was: Reopened) Unable to open webhdfs paths with + - Key: HDFS-7816 URL: https://issues.apache.org/jira/browse/HDFS-7816 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.7.0 Reporter: Jason Lowe Assignee: Kihwal Lee Priority: Blocker Attachments: HDFS-7816.patch webhdfs requests to open files with % characters in the filename fail because the filename is not being decoded properly. For example: $ hadoop fs -cat 'webhdfs://nn/user/somebody/abc%def' cat: File does not exist: /user/somebody/abc%25def -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7693) libhdfs: add hdfsFile cache
[ https://issues.apache.org/jira/browse/HDFS-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-7693: --- Resolution: Won't Fix Status: Resolved (was: Patch Available) libhdfs: add hdfsFile cache --- Key: HDFS-7693 URL: https://issues.apache.org/jira/browse/HDFS-7693 Project: Hadoop HDFS Issue Type: Improvement Components: libhdfs Affects Versions: 2.7.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-7693.001.patch Add an hdfsFile cache inside libhdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7806) Refactor: move StorageType.java from hadoop-hdfs to hadoop-common
[ https://issues.apache.org/jira/browse/HDFS-7806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329659#comment-14329659 ] Chris Nauroth commented on HDFS-7806: - That looks like a flaky Jenkins run. I submitted a new run here: https://builds.apache.org/job/PreCommit-HDFS-Build/9632/ Refactor: move StorageType.java from hadoop-hdfs to hadoop-common - Key: HDFS-7806 URL: https://issues.apache.org/jira/browse/HDFS-7806 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Priority: Minor Attachments: HDFS-7806.00.patch, HDFS-7806.01.patch, HDFS-7806.02.patch To report per storage type quota and usage information from hadoop fs -count -q or hdfs dfs -count -q, we need to migrate the StorageType definition from hadoop-hdfs (org.apache.hadoop.hdfs) to hadoop-common(org.apache.hadoop.fs) because the ContentSummary and FileSystem#getContentSummary() are in org.apache.hadoop.fs package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7816) Unable to open webhdfs paths with +
[ https://issues.apache.org/jira/browse/HDFS-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-7816: - Summary: Unable to open webhdfs paths with + (was: Unable to open webhdfs paths with escape characters) Unable to open webhdfs paths with + - Key: HDFS-7816 URL: https://issues.apache.org/jira/browse/HDFS-7816 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.7.0 Reporter: Jason Lowe Priority: Blocker webhdfs requests to open files with % characters in the filename fail because the filename is not being decoded properly. For example: $ hadoop fs -cat 'webhdfs://nn/user/somebody/abc%def' cat: File does not exist: /user/somebody/abc%25def -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7773) Additional metrics in HDFS to be accessed via jmx.
[ https://issues.apache.org/jira/browse/HDFS-7773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HDFS-7773. - Resolution: Fixed Fix Version/s: 2.7.0 +1 for the both the trunk and branch-2 patches. I have finished committing these. Anu, thank you for working on these new metrics. Additional metrics in HDFS to be accessed via jmx. -- Key: HDFS-7773 URL: https://issues.apache.org/jira/browse/HDFS-7773 Project: Hadoop HDFS Issue Type: Improvement Components: datanode, namenode Reporter: Anu Engineer Assignee: Anu Engineer Fix For: 2.7.0 Attachments: hdfs-7773.001.patch, hdfs-7773.002.patch, hdfs-7773.003.patch, hdfs-7773.branch-2.001.patch We would like to have the following metrics added to DataNode and name node this to improve Ambari dashboard 1) DN disk i/o utilization 2) DN network i/o utilization 3) Namenode read operations 4) Namenode write operations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7816) Unable to open webhdfs paths with +
[
https://issues.apache.org/jira/browse/HDFS-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329552#comment-14329552
]
Hadoop QA commented on HDFS-7816:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12699933/HDFS-7816.patch
against trunk revision 02e7dec.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9629//console
This message is automatically generated.
Unable to open webhdfs paths with +
-
Key: HDFS-7816
URL: https://issues.apache.org/jira/browse/HDFS-7816
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.7.0
Reporter: Jason Lowe
Assignee: Kihwal Lee
Priority: Blocker
Attachments: HDFS-7816.patch
webhdfs requests to open files with % characters in the filename fail because
the filename is not being decoded properly. For example:
$ hadoop fs -cat 'webhdfs://nn/user/somebody/abc%def'
cat: File does not exist: /user/somebody/abc%25def
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7807) libhdfs htable.c: fix htable resizing, add unit test
[ https://issues.apache.org/jira/browse/HDFS-7807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-7807: --- Attachment: HDFS-7807.002.patch libhdfs htable.c: fix htable resizing, add unit test Key: HDFS-7807 URL: https://issues.apache.org/jira/browse/HDFS-7807 Project: Hadoop HDFS Issue Type: Bug Components: native Affects Versions: 2.7.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-7807.001.patch, HDFS-7807.002.patch libhdfs htable.c: fix htable resizing, add unit test -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7769) TestHDFSCLI create files in hdfs project root dir
[ https://issues.apache.org/jira/browse/HDFS-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329696#comment-14329696 ] Konstantin Shvachko commented on HDFS-7769: --- [~chris.douglas] you probably have an opinion on whether a patch can be committed solely based on a non-committer plus one. I always thought a commiter's plus one is *required*, while Nicholas implies *any* plus one would do, citing our bylaws, which are vague on the matter. TestHDFSCLI create files in hdfs project root dir - Key: HDFS-7769 URL: https://issues.apache.org/jira/browse/HDFS-7769 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze Priority: Trivial Fix For: 2.7.0 Attachments: h7769_20150210.patch, h7769_20150210b.patch After running TestHDFSCLI, two files (data and .data.crc) remain in hdfs project root dir. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7807) libhdfs htable.c: fix htable resizing, add unit test
[ https://issues.apache.org/jira/browse/HDFS-7807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-7807: Hadoop Flags: Reviewed +1 for patch 002 pending Jenkins. I verified that the compiler warnings on Windows are gone. Thanks, Colin! libhdfs htable.c: fix htable resizing, add unit test Key: HDFS-7807 URL: https://issues.apache.org/jira/browse/HDFS-7807 Project: Hadoop HDFS Issue Type: Bug Components: native Affects Versions: 2.7.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-7807.001.patch, HDFS-7807.002.patch libhdfs htable.c: fix htable resizing, add unit test -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7188) support build libhdfs3 on windows
[ https://issues.apache.org/jira/browse/HDFS-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329528#comment-14329528 ] Thanh Do commented on HDFS-7188: Hi [~cmccabe]. Do you plan to take a closer look at my current patch or you will wait for the next patch? I prefer the former plan because I can integrate as many comments as possible for the next iteration. support build libhdfs3 on windows - Key: HDFS-7188 URL: https://issues.apache.org/jira/browse/HDFS-7188 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Environment: Windows System, Visual Studio 2010 Reporter: Zhanwei Wang Assignee: Thanh Do Attachments: HDFS-7188-branch-HDFS-6994-0.patch, HDFS-7188-branch-HDFS-6994-1.patch, HDFS-7188-branch-HDFS-6994-2.patch libhdfs3 should work on windows -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7773) Additional metrics in HDFS to be accessed via jmx.
[ https://issues.apache.org/jira/browse/HDFS-7773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329536#comment-14329536 ] Hudson commented on HDFS-7773: -- SUCCESS: Integrated in Hadoop-trunk-Commit #7166 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7166/]) HDFS-7773. Additional metrics in HDFS to be accessed via jmx. Contributed by Anu Engineer. (cnauroth: rev 02e7dec79d2d4f2b801435343219d8fb53ec931f) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java * hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java Additional metrics in HDFS to be accessed via jmx. -- Key: HDFS-7773 URL: https://issues.apache.org/jira/browse/HDFS-7773 Project: Hadoop HDFS Issue Type: Improvement Components: datanode, namenode Reporter: Anu Engineer Assignee: Anu Engineer Fix For: 2.7.0 Attachments: hdfs-7773.001.patch, hdfs-7773.002.patch, hdfs-7773.003.patch, hdfs-7773.branch-2.001.patch We would like to have the following metrics added to DataNode and name node this to improve Ambari dashboard 1) DN disk i/o utilization 2) DN network i/o utilization 3) Namenode read operations 4) Namenode write operations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7816) Unable to open webhdfs paths with +
[ https://issues.apache.org/jira/browse/HDFS-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-7816: - Attachment: HDFS-7816.patch Attaching a patch with the updated test case. The description and the link have also been updated. Unable to open webhdfs paths with + - Key: HDFS-7816 URL: https://issues.apache.org/jira/browse/HDFS-7816 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.7.0 Reporter: Jason Lowe Assignee: Kihwal Lee Priority: Blocker Attachments: HDFS-7816.patch webhdfs requests to open files with % characters in the filename fail because the filename is not being decoded properly. For example: $ hadoop fs -cat 'webhdfs://nn/user/somebody/abc%def' cat: File does not exist: /user/somebody/abc%25def -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7693) libhdfs: add hdfsFile cache
[ https://issues.apache.org/jira/browse/HDFS-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329585#comment-14329585 ] Colin Patrick McCabe commented on HDFS-7693: Thanks, Chris. Those are good questions. The motivation is to reduce NN RPC traffic in a distributed database. Partly the idea is if multiple queries use the same streams, we'd like to just keep them open rather than re-opening (and generating another getBlockLocations call on the NN). I did consider putting caching into hdfsOpen / hdfsClose, but I feel they're better off separate. Partly the nightmare of the FileSystem cache convinced me that hidden caches are usually bad. Partly it's just that I'd like to be able to evolve the cache in the future independently of hdfsOpen (like adding new eviction policies, etc.). I'm going to close this out since we decided to do the caching in an upper (application) layer rather than at the libhdfs level. It makes more sense there since the application (Impala) already has some APIs for cache invalidation... something we don't really know about at the libhdfs level. The cache is also somewhat logically separate from the rest of the libhdfs stuff, so it would be nice to have separation of concerns. libhdfs: add hdfsFile cache --- Key: HDFS-7693 URL: https://issues.apache.org/jira/browse/HDFS-7693 Project: Hadoop HDFS Issue Type: Improvement Components: libhdfs Affects Versions: 2.7.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-7693.001.patch Add an hdfsFile cache inside libhdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7740) Test truncate with DataNodes restarting
[
https://issues.apache.org/jira/browse/HDFS-7740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329683#comment-14329683
]
Konstantin Shvachko commented on HDFS-7740:
---
This actually worked pretty good.
Only one test is running too long:
{{testTruncateWithDataNodesShutdownImmediately()}} adds 30 secs to the running
time because you check {{isUnderConstruction()}} for the block 300 times with
interval 100 msec. I'd suggest waiting for DNs being down, and then checking
the block being still under construction.
{code}
cluster.shutdownDataNodes();
try {
for(int i = 0; i SUCCESS_ATTEMPTS cluster.isDataNodeUp(); i++) {
Thread.sleep(SLEEP);
}
assertFalse(All DataNodes should be down., cluster.isDataNodeUp());
LocatedBlocks blocks = getLocatedBlocks(p);
assertTrue(blocks.isUnderConstruction());
} finally {
cluster.startDataNodes(conf, DATANODE_NUM, true,
StartupOption.REGULAR, null);
cluster.waitActive();
}
{code}
I am +1 on the rest.
Yi, if you cannot update the patch I can make just this change to the latest
and commit.
Test truncate with DataNodes restarting
---
Key: HDFS-7740
URL: https://issues.apache.org/jira/browse/HDFS-7740
Project: Hadoop HDFS
Issue Type: Sub-task
Components: test
Affects Versions: 2.7.0
Reporter: Konstantin Shvachko
Assignee: Yi Liu
Fix For: 2.7.0
Attachments: HDFS-7740.001.patch, HDFS-7740.002.patch
Add a test case, which ensures replica consistency when DNs are failing and
restarting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7807) libhdfs htable.c: fix htable resizing, add unit test
[ https://issues.apache.org/jira/browse/HDFS-7807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329572#comment-14329572 ] Colin Patrick McCabe commented on HDFS-7807: Thanks for the review, Chris. bq. round_up_to_power_of_2 does not return a power of 2 for i == 0. However, we don't need it to, because we separately enforce HTABLE_MIN_SIZE of 4. Maybe just drop a quick comment here to prevent confusion? Hmm, good call. Let me just add a special case for i == 0 so that the function does what it says. Always good to keep things simple, and this doesn't need to be super-optimized since it only runs once. :) bq. On Windows, the compiler spews a bunch of type cast and loss of precision warnings on the test code. This is really just a side effect of manually setting void pointer values to stub the keys and values for the tests. I think it's acceptable to stifle the warnings by adding the following right before main: thanks, added libhdfs htable.c: fix htable resizing, add unit test Key: HDFS-7807 URL: https://issues.apache.org/jira/browse/HDFS-7807 Project: Hadoop HDFS Issue Type: Bug Components: native Affects Versions: 2.7.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-7807.001.patch libhdfs htable.c: fix htable resizing, add unit test -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7816) Unable to open webhdfs paths with +
[
https://issues.apache.org/jira/browse/HDFS-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329611#comment-14329611
]
Haohui Mai commented on HDFS-7816:
--
My impression is that {{WebHdfsFileSystem}} is passing the wrong url, thus it
looks like the fixes should be done in {{WebHdfsFileSystem}}.
Unable to open webhdfs paths with +
-
Key: HDFS-7816
URL: https://issues.apache.org/jira/browse/HDFS-7816
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.7.0
Reporter: Jason Lowe
Assignee: Kihwal Lee
Priority: Blocker
Attachments: HDFS-7816.patch, HDFS-7816.patch
webhdfs requests to open files with % characters in the filename fail because
the filename is not being decoded properly. For example:
$ hadoop fs -cat 'webhdfs://nn/user/somebody/abc%def'
cat: File does not exist: /user/somebody/abc%25def
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7807) libhdfs htable.c: fix htable resizing, add unit test
[
https://issues.apache.org/jira/browse/HDFS-7807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329859#comment-14329859
]
Hadoop QA commented on HDFS-7807:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12699938/HDFS-7807.002.patch
against trunk revision 3c5ff07.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9630//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9630//console
This message is automatically generated.
libhdfs htable.c: fix htable resizing, add unit test
Key: HDFS-7807
URL: https://issues.apache.org/jira/browse/HDFS-7807
Project: Hadoop HDFS
Issue Type: Bug
Components: native
Affects Versions: 2.7.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-7807.001.patch, HDFS-7807.002.patch
libhdfs htable.c: fix htable resizing, add unit test
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7816) Unable to open webhdfs paths with +
[
https://issues.apache.org/jira/browse/HDFS-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329869#comment-14329869
]
Hadoop QA commented on HDFS-7816:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12699945/HDFS-7816.patch
against trunk revision 3c5ff07.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestFileTruncate
org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9631//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9631//console
This message is automatically generated.
Unable to open webhdfs paths with +
-
Key: HDFS-7816
URL: https://issues.apache.org/jira/browse/HDFS-7816
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.7.0
Reporter: Jason Lowe
Assignee: Kihwal Lee
Priority: Blocker
Attachments: HDFS-7816.patch, HDFS-7816.patch
webhdfs requests to open files with % characters in the filename fail because
the filename is not being decoded properly. For example:
$ hadoop fs -cat 'webhdfs://nn/user/somebody/abc%def'
cat: File does not exist: /user/somebody/abc%25def
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7806) Refactor: move StorageType.java from hadoop-hdfs to hadoop-common
[ https://issues.apache.org/jira/browse/HDFS-7806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329941#comment-14329941 ] Xiaoyu Yao commented on HDFS-7806: -- TestDFSHAAdminMiniCluster#testFencer should have been fixed by HDFS-7813. Refactor: move StorageType.java from hadoop-hdfs to hadoop-common - Key: HDFS-7806 URL: https://issues.apache.org/jira/browse/HDFS-7806 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Priority: Minor Attachments: HDFS-7806.00.patch, HDFS-7806.01.patch, HDFS-7806.02.patch To report per storage type quota and usage information from hadoop fs -count -q or hdfs dfs -count -q, we need to migrate the StorageType definition from hadoop-hdfs (org.apache.hadoop.hdfs) to hadoop-common(org.apache.hadoop.fs) because the ContentSummary and FileSystem#getContentSummary() are in org.apache.hadoop.fs package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7491) Add incremental blockreport latency to DN metrics
[ https://issues.apache.org/jira/browse/HDFS-7491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7491: -- Attachment: HDFS-7491-branch-2.patch HDFS-7491-3.patch Thanks, Chris. Here are the updated patches for trunk and branch-2. Add incremental blockreport latency to DN metrics - Key: HDFS-7491 URL: https://issues.apache.org/jira/browse/HDFS-7491 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Reporter: Ming Ma Assignee: Ming Ma Priority: Minor Attachments: HDFS-7491-2.patch, HDFS-7491-3.patch, HDFS-7491-branch-2.patch, HDFS-7491.patch In a busy cluster, IBR processing could be delayed due to NN FSNamesystem lock and cause NN to throw NotReplicatedYetException to DFSClient and thus increase the overall application latency. This will be taken care of when we address the NN FSNamesystem lock contention issue. It is useful if we can provide IBR latency metrics from DN's point of view. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6133) Make Balancer support exclude specified path
[ https://issues.apache.org/jira/browse/HDFS-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14330028#comment-14330028 ] Yongjun Zhang commented on HDFS-6133: - Hi Nicholas, Assuming that user doesn't change config during the upgrade process, since by default pinning is disabled, I guess your suggestion may be fine. Would you please comment on whether the assumption that user doesn't change config is a valid assumption? It seems we don't have a hard control on that, which is what I was a little worried. Maybe we can change the config description to emphasize this rule? Thanks. Make Balancer support exclude specified path Key: HDFS-6133 URL: https://issues.apache.org/jira/browse/HDFS-6133 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover, datanode Reporter: zhaoyunjiong Assignee: zhaoyunjiong Fix For: 2.7.0 Attachments: HDFS-6133-1.patch, HDFS-6133-10.patch, HDFS-6133-11.patch, HDFS-6133-2.patch, HDFS-6133-3.patch, HDFS-6133-4.patch, HDFS-6133-5.patch, HDFS-6133-6.patch, HDFS-6133-7.patch, HDFS-6133-8.patch, HDFS-6133-9.patch, HDFS-6133.patch Currently, run Balancer will destroying Regionserver's data locality. If getBlocks could exclude blocks belongs to files which have specific path prefix, like /hbase, then we can run Balancer without destroying Regionserver's data locality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7807) libhdfs htable.c: fix htable resizing, add unit test
[
https://issues.apache.org/jira/browse/HDFS-7807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329860#comment-14329860
]
Chris Nauroth commented on HDFS-7807:
-
{{TestDFSHAAdminMiniCluster}} is an unrelated known test failure tracked in
HDFS-7813.
libhdfs htable.c: fix htable resizing, add unit test
Key: HDFS-7807
URL: https://issues.apache.org/jira/browse/HDFS-7807
Project: Hadoop HDFS
Issue Type: Bug
Components: native
Affects Versions: 2.7.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-7807.001.patch, HDFS-7807.002.patch
libhdfs htable.c: fix htable resizing, add unit test
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7813) TestDFSHAAdminMiniCluster#testFencer testcase is failing frequently
[
https://issues.apache.org/jira/browse/HDFS-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-7813:
Target Version/s: 2.7.0
TestDFSHAAdminMiniCluster#testFencer testcase is failing frequently
---
Key: HDFS-7813
URL: https://issues.apache.org/jira/browse/HDFS-7813
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, test
Reporter: Rakesh R
Assignee: Rakesh R
Attachments: HDFS-7813-001.patch
Following is the failure trace.
{code}
java.lang.AssertionError: expected:0 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster.testFencer(TestDFSHAAdminMiniCluster.java:163)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7806) Refactor: move StorageType.java from hadoop-hdfs to hadoop-common
[ https://issues.apache.org/jira/browse/HDFS-7806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329956#comment-14329956 ] Arpit Agarwal commented on HDFS-7806: - +1 for the v02 patch. Will hold off committing in case [~cnauroth] has any feedback. Refactor: move StorageType.java from hadoop-hdfs to hadoop-common - Key: HDFS-7806 URL: https://issues.apache.org/jira/browse/HDFS-7806 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Priority: Minor Attachments: HDFS-7806.00.patch, HDFS-7806.01.patch, HDFS-7806.02.patch To report per storage type quota and usage information from hadoop fs -count -q or hdfs dfs -count -q, we need to migrate the StorageType definition from hadoop-hdfs (org.apache.hadoop.hdfs) to hadoop-common(org.apache.hadoop.fs) because the ContentSummary and FileSystem#getContentSummary() are in org.apache.hadoop.fs package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7816) Unable to open webhdfs paths with +
[ https://issues.apache.org/jira/browse/HDFS-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329790#comment-14329790 ] Jason Lowe commented on HDFS-7816: -- I don't think we can rely on clients changing the way the URL is encoded, otherwise we break compatibility with older clients. I think Kihwal's patch will work even with older clients. My main concern is that we're relying on QueryStringDecoder#path to give us a raw path so URI can decode it properly. The javadoc for that method says it returns a decoded path, and if that were ever fixed to match the javadoc then we'd end up double-decoding which will break for some paths. It also seems weird to me that we're using a QueryStringDecoder to obtain parts of the URL that aren't query strings. I think it would be safer to avoid QueryStringDecoder altogether for the path computation and just pass the original request URI string to the URI constructor for path decoding. Unable to open webhdfs paths with + - Key: HDFS-7816 URL: https://issues.apache.org/jira/browse/HDFS-7816 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.7.0 Reporter: Jason Lowe Assignee: Kihwal Lee Priority: Blocker Attachments: HDFS-7816.patch, HDFS-7816.patch webhdfs requests to open files with % characters in the filename fail because the filename is not being decoded properly. For example: $ hadoop fs -cat 'webhdfs://nn/user/somebody/abc%def' cat: File does not exist: /user/somebody/abc%25def -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7009) Active NN and standby NN have different live nodes
[
https://issues.apache.org/jira/browse/HDFS-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329819#comment-14329819
]
Chris Nauroth commented on HDFS-7009:
-
Thanks for updating the patch, Ming. I wasn't thinking of a fix at the RPC
client layer, but after seeing the patch, I think this is the right thing to
do. The protobuf {{parseDelimitedFrom}} method is documented to return
{{null}} if the input stream is already at EOF, so semantically,
{{EOFException}} is the right error code. This change may also benefit other
RPC clients, such as YARN's {{RMProxy}}, where there is a retry policy
associated with {{EOFException}}.
Since this is a change lower down at the RPC layer, I'd like to wait until next
week to commit, in case anyone else wants to review. I'm also notifying
[~szetszwo], who originally worked on this code for HDFS-3504 (configurable
retry policies for DFSClient). Nicholas, do you see any problem with making
this change?
You'll need to update the patch one more time. The method signature of
{{sendHeartbeat}} changed recently. You'll need to add one more parameter to
that call in the test, and it can be set to
{{Mockito.any(VolumeFailureSummary.class)}}. There are also some typos:
mokito instead of mockito. Let's correct those.
The test failure in the last Jenkins run appears to be unrelated.
Thanks again for your work on this, Ming!
Active NN and standby NN have different live nodes
--
Key: HDFS-7009
URL: https://issues.apache.org/jira/browse/HDFS-7009
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Ming Ma
Assignee: Ming Ma
Attachments: HDFS-7009-2.patch, HDFS-7009-3.patch, HDFS-7009.patch
To follow up on https://issues.apache.org/jira/browse/HDFS-6478, in most
cases, given DN sends HB and BR to NN regularly, if a specific RPC call
fails, it isn't a big deal.
However, there are cases where DN fails to register with NN during initial
handshake due to exceptions not covered by RPC client's connection retry.
When this happens, the DN won't talk to that NN until the DN restarts.
{noformat}
BPServiceActor
public void run() {
LOG.info(this + starting to offer service);
try {
// init stuff
try {
// setup storage
connectToNNAndHandshake();
} catch (IOException ioe) {
// Initial handshake, storage recovery or registration failed
// End BPOfferService thread
LOG.fatal(Initialization failed for block pool + this, ioe);
return;
}
initialized = true; // bp is initialized;
while (shouldRun()) {
try {
offerService();
} catch (Exception ex) {
LOG.error(Exception in BPOfferService for + this, ex);
sleepAndLogInterrupts(5000, offering service);
}
}
...
{noformat}
Here is an example of the call stack.
{noformat}
java.io.IOException: Failed on local exception: java.io.IOException: Response
is null.; Host Details : local host is: xxx; destination host is:
yyy:8030;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:761)
at org.apache.hadoop.ipc.Client.call(Client.java:1239)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
at com.sun.proxy.$Proxy9.registerDatanode(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
at com.sun.proxy.$Proxy9.registerDatanode(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.registerDatanode(DatanodeProtocolClientSideTranslatorPB.java:146)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:623)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Response is null.
at
org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:949)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:844)
{noformat}
This will create discrepancy between active NN and standby NN in terms
[jira] [Commented] (HDFS-7817) libhdfs3: fix strerror_r detection
[
https://issues.apache.org/jira/browse/HDFS-7817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329873#comment-14329873
]
Thanh Do commented on HDFS-7817:
Hey [~cmccabe], can you point me to the {{terror}} in libhdfs? I greped for
this name but couldn't find it. I would like to take a crack at this if you
don't mind. Thanks!
libhdfs3: fix strerror_r detection
--
Key: HDFS-7817
URL: https://issues.apache.org/jira/browse/HDFS-7817
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Reporter: Colin Patrick McCabe
The signature of strerror_r is not quite detected correctly in libhdfs3. The
code assumes that {{int foo = strerror_r}} will fail to compile with the GNU
type signature, but this is not the case (C\+\+ will coerce the char* to an
int in this case). Instead, we should do what the libhdfs {{terror}}
(threaded error) function does here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7813) TestDFSHAAdminMiniCluster#testFencer testcase is failing frequently
[
https://issues.apache.org/jira/browse/HDFS-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-7813:
Summary: TestDFSHAAdminMiniCluster#testFencer testcase is failing
frequently (was: TestDFSHAAdminMiniCluster#testFencer testcase is failing
ferquently)
TestDFSHAAdminMiniCluster#testFencer testcase is failing frequently
---
Key: HDFS-7813
URL: https://issues.apache.org/jira/browse/HDFS-7813
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, test
Reporter: Rakesh R
Assignee: Rakesh R
Attachments: HDFS-7813-001.patch
Following is the failure trace.
{code}
java.lang.AssertionError: expected:0 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster.testFencer(TestDFSHAAdminMiniCluster.java:163)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7769) TestHDFSCLI create files in hdfs project root dir
[ https://issues.apache.org/jira/browse/HDFS-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329890#comment-14329890 ] Chris Douglas commented on HDFS-7769: - As written, the bylaws require a +1 from a committer. It's been enforced fairly strictly in MapReduce/YARN. The language in the bylaws is pretty direct: bq. Consensus approval of active committers, but with a minimum of one +1 The ambiguity is whether the contributor can self-approve patches, or (unwritten) +1 a trivial patch with any reviewer's approval? I'd say no, if only to avoid arguments over what constitutes a trivial patch. bq. it does not quite make sense for non-committers reviewing patches. As least, no one has incentive to ask a non-committers to reviewing a patch. Non-committers review patches to provide feedback on them and signal that the patch has already had a pair of eyes on it. I'm more likely to +1 a patch if someone else has done a pass on it and provided good feedback. Requiring another committer to review also blocks development on dead areas of the codebase with only a single maintainer. Those either need to move out of the project or find new committers to help maintain it. TestHDFSCLI create files in hdfs project root dir - Key: HDFS-7769 URL: https://issues.apache.org/jira/browse/HDFS-7769 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze Priority: Trivial Fix For: 2.7.0 Attachments: h7769_20150210.patch, h7769_20150210b.patch After running TestHDFSCLI, two files (data and .data.crc) remain in hdfs project root dir. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7806) Refactor: move StorageType.java from hadoop-hdfs to hadoop-common
[
https://issues.apache.org/jira/browse/HDFS-7806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329911#comment-14329911
]
Hadoop QA commented on HDFS-7806:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12699785/HDFS-7806.02.patch
against trunk revision f56c65b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 34 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9632//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9632//console
This message is automatically generated.
Refactor: move StorageType.java from hadoop-hdfs to hadoop-common
-
Key: HDFS-7806
URL: https://issues.apache.org/jira/browse/HDFS-7806
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Priority: Minor
Attachments: HDFS-7806.00.patch, HDFS-7806.01.patch,
HDFS-7806.02.patch
To report per storage type quota and usage information from hadoop fs -count
-q or hdfs dfs -count -q, we need to migrate the StorageType definition
from hadoop-hdfs (org.apache.hadoop.hdfs) to
hadoop-common(org.apache.hadoop.fs) because the ContentSummary and
FileSystem#getContentSummary() are in org.apache.hadoop.fs package.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7495) Lock inversion in DFSInputStream#getBlockAt()
[
https://issues.apache.org/jira/browse/HDFS-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HDFS-7495:
-
Resolution: Later
Status: Resolved (was: Patch Available)
Lock inversion in DFSInputStream#getBlockAt()
-
Key: HDFS-7495
URL: https://issues.apache.org/jira/browse/HDFS-7495
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Attachments: hdfs-7495-001.patch
There're two locks: one on DFSInputStream.this , one on
DFSInputStream.infoLock
Normally lock is obtained on infoLock, then on DFSInputStream.infoLock
However, such order is not observed in DFSInputStream#getBlockAt() :
{code}
synchronized(infoLock) {
...
if (updatePosition) {
// synchronized not strictly needed, since we only get here
// from synchronized caller methods
synchronized(this) {
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7491) Add incremental blockreport latency to DN metrics
[
https://issues.apache.org/jira/browse/HDFS-7491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329995#comment-14329995
]
Hadoop QA commented on HDFS-7491:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/1273/HDFS-7491-branch-2.patch
against trunk revision 6f01330.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9634//console
This message is automatically generated.
Add incremental blockreport latency to DN metrics
-
Key: HDFS-7491
URL: https://issues.apache.org/jira/browse/HDFS-7491
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Reporter: Ming Ma
Assignee: Ming Ma
Priority: Minor
Attachments: HDFS-7491-2.patch, HDFS-7491-3.patch,
HDFS-7491-branch-2.patch, HDFS-7491.patch
In a busy cluster, IBR processing could be delayed due to NN FSNamesystem
lock and cause NN to throw NotReplicatedYetException to DFSClient and thus
increase the overall application latency.
This will be taken care of when we address the NN FSNamesystem lock
contention issue.
It is useful if we can provide IBR latency metrics from DN's point of view.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7813) TestDFSHAAdminMiniCluster#testFencer testcase is failing frequently
[
https://issues.apache.org/jira/browse/HDFS-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329886#comment-14329886
]
Hudson commented on HDFS-7813:
--
FAILURE: Integrated in Hadoop-trunk-Commit #7170 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7170/])
HDFS-7813. TestDFSHAAdminMiniCluster#testFencer testcase is failing frequently.
Contributed by Rakesh R. (cnauroth: rev
0d6af574e0056fc627461eb91ed0c365b026b470)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java
TestDFSHAAdminMiniCluster#testFencer testcase is failing frequently
---
Key: HDFS-7813
URL: https://issues.apache.org/jira/browse/HDFS-7813
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, test
Reporter: Rakesh R
Assignee: Rakesh R
Fix For: 2.7.0
Attachments: HDFS-7813-001.patch
Following is the failure trace.
{code}
java.lang.AssertionError: expected:0 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster.testFencer(TestDFSHAAdminMiniCluster.java:163)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7816) Unable to open webhdfs paths with +
[ https://issues.apache.org/jira/browse/HDFS-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329935#comment-14329935 ] Brahma Reddy Battula commented on HDFS-7816: Testcase failures are unrelated to this patch.. Unable to open webhdfs paths with + - Key: HDFS-7816 URL: https://issues.apache.org/jira/browse/HDFS-7816 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.7.0 Reporter: Jason Lowe Assignee: Kihwal Lee Priority: Blocker Attachments: HDFS-7816.patch, HDFS-7816.patch webhdfs requests to open files with % characters in the filename fail because the filename is not being decoded properly. For example: $ hadoop fs -cat 'webhdfs://nn/user/somebody/abc%def' cat: File does not exist: /user/somebody/abc%25def -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7009) Active NN and standby NN have different live nodes
[
https://issues.apache.org/jira/browse/HDFS-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated HDFS-7009:
--
Attachment: HDFS-7009-4.patch
Thanks, Chris. Here is the updated patch.
Nicholas can confirm, {{FailoverOnNetworkExceptionRetry}} defined in
{{RetryPolicies}} handles {{IOException}} that isn't {{RemoteException}}. So
this change shouldn't change that behavior.
Active NN and standby NN have different live nodes
--
Key: HDFS-7009
URL: https://issues.apache.org/jira/browse/HDFS-7009
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Ming Ma
Assignee: Ming Ma
Attachments: HDFS-7009-2.patch, HDFS-7009-3.patch, HDFS-7009-4.patch,
HDFS-7009.patch
To follow up on https://issues.apache.org/jira/browse/HDFS-6478, in most
cases, given DN sends HB and BR to NN regularly, if a specific RPC call
fails, it isn't a big deal.
However, there are cases where DN fails to register with NN during initial
handshake due to exceptions not covered by RPC client's connection retry.
When this happens, the DN won't talk to that NN until the DN restarts.
{noformat}
BPServiceActor
public void run() {
LOG.info(this + starting to offer service);
try {
// init stuff
try {
// setup storage
connectToNNAndHandshake();
} catch (IOException ioe) {
// Initial handshake, storage recovery or registration failed
// End BPOfferService thread
LOG.fatal(Initialization failed for block pool + this, ioe);
return;
}
initialized = true; // bp is initialized;
while (shouldRun()) {
try {
offerService();
} catch (Exception ex) {
LOG.error(Exception in BPOfferService for + this, ex);
sleepAndLogInterrupts(5000, offering service);
}
}
...
{noformat}
Here is an example of the call stack.
{noformat}
java.io.IOException: Failed on local exception: java.io.IOException: Response
is null.; Host Details : local host is: xxx; destination host is:
yyy:8030;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:761)
at org.apache.hadoop.ipc.Client.call(Client.java:1239)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
at com.sun.proxy.$Proxy9.registerDatanode(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
at com.sun.proxy.$Proxy9.registerDatanode(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.registerDatanode(DatanodeProtocolClientSideTranslatorPB.java:146)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:623)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Response is null.
at
org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:949)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:844)
{noformat}
This will create discrepancy between active NN and standby NN in terms of
live nodes.
Here is a possible scenario of missing blocks after failover.
1. DN A, B set up handshakes with active NN, but not with standby NN.
2. A block is replicated to DN A, B and C.
3. From standby NN's point of view, given A and B are dead nodes, the block
is under replicated.
4. DN C is down.
5. Before active NN detects DN C is down, it fails over.
6. The new active NN considers the block is missing. Even though there are
two replicas on DN A and B.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7788) Post-2.6 namenode may not start up with an image containing inodes created with an old release.
[ https://issues.apache.org/jira/browse/HDFS-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329025#comment-14329025 ] Kihwal Lee commented on HDFS-7788: -- After a git pull, TestDFSHAAdminMiniCluster#testFencer failing consistently. It looks like HDFS-7813, unrelated this change. I will review HDFS-7813. Post-2.6 namenode may not start up with an image containing inodes created with an old release. --- Key: HDFS-7788 URL: https://issues.apache.org/jira/browse/HDFS-7788 Project: Hadoop HDFS Issue Type: Bug Reporter: Kihwal Lee Assignee: Rushabh S Shah Priority: Blocker Attachments: HDFS-7788-binary.patch, rushabh.patch Before HDFS-4305, which was fixed in 2.1.0-beta, clients could specify arbitrarily small preferred block size for a file including 0. This was normally done by faulty clients or failed creates, but it was possible. Until 2.5, reading a fsimage containing inodes with 0 byte preferred block size was allowed. So if a fsimage contained such an inode, the namenode would come up fine. In 2.6, the preferred block size is required be 0. Because of this change, the image that worked with 2.5 may not work with 2.6. If a cluster ran a version of hadoop earlier than 2.1.0-beta before, it is under this risk even if it worked fine with 2.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7788) Post-2.6 namenode may not start up with an image containing inodes created with an old release.
[ https://issues.apache.org/jira/browse/HDFS-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-7788: - Resolution: Fixed Fix Version/s: 2.7.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've committed this to trunk and branch-2. Thanks for working on the fix, [~shahrs87]. Post-2.6 namenode may not start up with an image containing inodes created with an old release. --- Key: HDFS-7788 URL: https://issues.apache.org/jira/browse/HDFS-7788 Project: Hadoop HDFS Issue Type: Bug Reporter: Kihwal Lee Assignee: Rushabh S Shah Priority: Blocker Fix For: 2.7.0 Attachments: HDFS-7788-binary.patch, rushabh.patch Before HDFS-4305, which was fixed in 2.1.0-beta, clients could specify arbitrarily small preferred block size for a file including 0. This was normally done by faulty clients or failed creates, but it was possible. Until 2.5, reading a fsimage containing inodes with 0 byte preferred block size was allowed. So if a fsimage contained such an inode, the namenode would come up fine. In 2.6, the preferred block size is required be 0. Because of this change, the image that worked with 2.5 may not work with 2.6. If a cluster ran a version of hadoop earlier than 2.1.0-beta before, it is under this risk even if it worked fine with 2.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7788) Post-2.6 namenode may not start up with an image containing inodes created with an old release.
[ https://issues.apache.org/jira/browse/HDFS-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329036#comment-14329036 ] Hudson commented on HDFS-7788: -- FAILURE: Integrated in Hadoop-trunk-Commit #7161 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7161/]) HDFS-7788. Post-2.6 namenode may not start up with an image containing inodes created with an old release. Contributed by Rushabh Shah. (kihwal: rev 7ae5255a1613ccfb43646f33eabacf1062c86e93) * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/image-with-zero-block-size.tar.gz * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LongBitFormat.java Post-2.6 namenode may not start up with an image containing inodes created with an old release. --- Key: HDFS-7788 URL: https://issues.apache.org/jira/browse/HDFS-7788 Project: Hadoop HDFS Issue Type: Bug Reporter: Kihwal Lee Assignee: Rushabh S Shah Priority: Blocker Fix For: 2.7.0 Attachments: HDFS-7788-binary.patch, rushabh.patch Before HDFS-4305, which was fixed in 2.1.0-beta, clients could specify arbitrarily small preferred block size for a file including 0. This was normally done by faulty clients or failed creates, but it was possible. Until 2.5, reading a fsimage containing inodes with 0 byte preferred block size was allowed. So if a fsimage contained such an inode, the namenode would come up fine. In 2.6, the preferred block size is required be 0. Because of this change, the image that worked with 2.5 may not work with 2.6. If a cluster ran a version of hadoop earlier than 2.1.0-beta before, it is under this risk even if it worked fine with 2.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7788) Post-2.6 namenode may not start up with an image containing inodes created with an old release.
[ https://issues.apache.org/jira/browse/HDFS-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329067#comment-14329067 ] Rushabh S Shah commented on HDFS-7788: -- Thanks Kihwal for reviewing and committting !! Post-2.6 namenode may not start up with an image containing inodes created with an old release. --- Key: HDFS-7788 URL: https://issues.apache.org/jira/browse/HDFS-7788 Project: Hadoop HDFS Issue Type: Bug Reporter: Kihwal Lee Assignee: Rushabh S Shah Priority: Blocker Fix For: 2.7.0 Attachments: HDFS-7788-binary.patch, rushabh.patch Before HDFS-4305, which was fixed in 2.1.0-beta, clients could specify arbitrarily small preferred block size for a file including 0. This was normally done by faulty clients or failed creates, but it was possible. Until 2.5, reading a fsimage containing inodes with 0 byte preferred block size was allowed. So if a fsimage contained such an inode, the namenode would come up fine. In 2.6, the preferred block size is required be 0. Because of this change, the image that worked with 2.5 may not work with 2.6. If a cluster ran a version of hadoop earlier than 2.1.0-beta before, it is under this risk even if it worked fine with 2.5. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7435) PB encoding of block reports is very inefficient
[
https://issues.apache.org/jira/browse/HDFS-7435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329090#comment-14329090
]
Daryn Sharp commented on HDFS-7435:
---
Let me get back to this patch this afternoon. I'll post perf numbers too.
PB encoding of block reports is very inefficient
Key: HDFS-7435
URL: https://issues.apache.org/jira/browse/HDFS-7435
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, namenode
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
Attachments: HDFS-7435.000.patch, HDFS-7435.001.patch,
HDFS-7435.002.patch, HDFS-7435.patch
Block reports are encoded as a PB repeating long. Repeating fields use an
{{ArrayList}} with default capacity of 10. A block report containing tens or
hundreds of thousand of longs (3 for each replica) is extremely expensive
since the {{ArrayList}} must realloc many times. Also, decoding repeating
fields will box the primitive longs which must then be unboxed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7501) TransactionsSinceLastCheckpoint can be negative on SBNs
[ https://issues.apache.org/jira/browse/HDFS-7501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329098#comment-14329098 ] Harsh J commented on HDFS-7501: --- Hey [~ggop], can you switch the method to simply make use of FSImage.getLastAppliedOrWrittenTxId(…) method? That should work just fine for both ANN and SBN modes, as it appears to track both written mode edits and tailed mode edits. The test can be modified to be a test of 0 as opposed to = 0 as we previously targeted. TransactionsSinceLastCheckpoint can be negative on SBNs --- Key: HDFS-7501 URL: https://issues.apache.org/jira/browse/HDFS-7501 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.5.0 Reporter: Harsh J Assignee: Gautam Gopalakrishnan Priority: Trivial Attachments: HDFS-7501-2.patch, HDFS-7501.patch The metric TransactionsSinceLastCheckpoint is derived as FSEditLog.txid minus NNStorage.mostRecentCheckpointTxId. In Standby mode, the former does not increment beyond the loaded or last-when-active value, but the latter does change due to checkpoints done regularly in this mode. Thereby, the SBN will eventually end up showing negative values for TransactionsSinceLastCheckpoint. This is not an issue as the metric only makes sense to be monitored on the Active NameNode, but we should perhaps just show the value 0 by detecting if the NN is in SBN form, as allowing a negative number is confusing to view within a chart that tracks it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7435) PB encoding of block reports is very inefficient
[
https://issues.apache.org/jira/browse/HDFS-7435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329107#comment-14329107
]
Suresh Srinivas commented on HDFS-7435:
---
[~daryn], can you please comment on [~jingzhao]'s patch and why it does not
address the issue? It would be great to get this done early.
PB encoding of block reports is very inefficient
Key: HDFS-7435
URL: https://issues.apache.org/jira/browse/HDFS-7435
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, namenode
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
Attachments: HDFS-7435.000.patch, HDFS-7435.001.patch,
HDFS-7435.002.patch, HDFS-7435.patch
Block reports are encoded as a PB repeating long. Repeating fields use an
{{ArrayList}} with default capacity of 10. A block report containing tens or
hundreds of thousand of longs (3 for each replica) is extremely expensive
since the {{ArrayList}} must realloc many times. Also, decoding repeating
fields will box the primitive longs which must then be unboxed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7740) Test truncate with DataNodes restarting
[
https://issues.apache.org/jira/browse/HDFS-7740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14330055#comment-14330055
]
Yi Liu commented on HDFS-7740:
--
Thanks [~shv] for you review, your suggestion about
{{testTruncateWithDataNodesShutdownImmediately}} is pretty good and I will
update the patch for that and commit it later.
{quote}
I am +1 on the rest.
Yi, if you cannot update the patch I can make just this change to the latest
and commit.
{quote}
Thanks a lot :) I can get time to update and commit.
Test truncate with DataNodes restarting
---
Key: HDFS-7740
URL: https://issues.apache.org/jira/browse/HDFS-7740
Project: Hadoop HDFS
Issue Type: Sub-task
Components: test
Affects Versions: 2.7.0
Reporter: Konstantin Shvachko
Assignee: Yi Liu
Fix For: 2.7.0
Attachments: HDFS-7740.001.patch, HDFS-7740.002.patch
Add a test case, which ensures replica consistency when DNs are failing and
restarting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7805) NameNode recovery prompt should be printed on console
[
https://issues.apache.org/jira/browse/HDFS-7805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
surendra singh lilhore updated HDFS-7805:
-
Attachment: HDFS-7805_1.patch
NameNode recovery prompt should be printed on console
-
Key: HDFS-7805
URL: https://issues.apache.org/jira/browse/HDFS-7805
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
Attachments: HDFS-7805.patch, HDFS-7805_1.patch
In my cluster root logger is not console, so when I run namenode recovery
tool MetaRecoveryContext.java prompt message is logged in log file.
Actually is should be display on console.
Currently it is like this
{code}
LOG.info(prompt);
{code}
It should be
{code}
System.err.print(prompt);
{code}
NameNode recovery prompt should be printed on console
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-7815) Loop on 'blocks does not belong to any file'
[ https://issues.apache.org/jira/browse/HDFS-7815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HDFS-7815. - Resolution: Duplicate Loop on 'blocks does not belong to any file' Key: HDFS-7815 URL: https://issues.apache.org/jira/browse/HDFS-7815 Project: Hadoop HDFS Issue Type: Bug Components: datanode, namenode Affects Versions: 2.6.0 Environment: small cluster on RetHat. 2 namenodes (HA), 6 datanodes with 19TB disk for hdfs. Reporter: Frode Halvorsen I am currently experincing a looping situation; The namenode uses appx 1:50 (min:sec) to log a massive amount of lines stating that some blocks don't belong to any file. During this time, it's unresponsive to any requests from datanodes, and if the zoo-keper had been running, it would have taken the name-node down (ssh-fencing : kill). When it has finished the 'round', it starts to do some normal work, and among other things, telling the datanode to delete the blocks. But before the datanode has gotten around to delete the blocks, and is about to report back to the namenode, the namenode has stared on the next round of reporing the same blocks that don't belong to anly file. Thus, the datanode gets a timout when reporing block-updates for the deleted blocks, And this, of course repeats itself over and over again... There is actually two issues , I think,; 1- the namenode gets totally unresponsive when reporing the blocks (could this be a debug-line instead of a INFO-line) 2 - the namenode seems to 'forget' that it has already reported those blocks just 2-3 minutes ago... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7815) Loop on 'blocks does not belong to any file'
[ https://issues.apache.org/jira/browse/HDFS-7815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329249#comment-14329249 ] Chris Nauroth commented on HDFS-7815: - Hello, [~frha]. This bug was fixed in HDFS-7503 by moving this logging outside of the namesystem write lock, so even if there is a large volume of this logging, other NameNode threads can still make progress. The fix is targeted to Apache Hadoop 2.6.1 and 2.7.0, both still awaiting release. In the meantime, a known workaround is to edit log4j.properties to tune down the logger level to WARN. Of course, this will have the side effect of suppressing these log messages entirely. I'm resolving this issue as duplicate. Loop on 'blocks does not belong to any file' Key: HDFS-7815 URL: https://issues.apache.org/jira/browse/HDFS-7815 Project: Hadoop HDFS Issue Type: Bug Components: datanode, namenode Affects Versions: 2.6.0 Environment: small cluster on RetHat. 2 namenodes (HA), 6 datanodes with 19TB disk for hdfs. Reporter: Frode Halvorsen I am currently experincing a looping situation; The namenode uses appx 1:50 (min:sec) to log a massive amount of lines stating that some blocks don't belong to any file. During this time, it's unresponsive to any requests from datanodes, and if the zoo-keper had been running, it would have taken the name-node down (ssh-fencing : kill). When it has finished the 'round', it starts to do some normal work, and among other things, telling the datanode to delete the blocks. But before the datanode has gotten around to delete the blocks, and is about to report back to the namenode, the namenode has stared on the next round of reporing the same blocks that don't belong to anly file. Thus, the datanode gets a timout when reporing block-updates for the deleted blocks, And this, of course repeats itself over and over again... There is actually two issues , I think,; 1- the namenode gets totally unresponsive when reporing the blocks (could this be a debug-line instead of a INFO-line) 2 - the namenode seems to 'forget' that it has already reported those blocks just 2-3 minutes ago... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7816) Unable to open webhdfs paths with escape characters
[ https://issues.apache.org/jira/browse/HDFS-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329199#comment-14329199 ] Jason Lowe commented on HDFS-7816: -- This appears to be broken after HDFS-7279. What's happening is the URI is being properly encoded by the client but the datanode is not decoding it properly when it tries to act as a DFS client. Looks like netty's QueryStringDecoder#path method, despite the javadoc stating it returns a decoded path, is not returning a decoded path from the URI. It doesn't use a URI object to decode it, rather it just performs substrings on the URI string. Even if you pass it a URI it uses the raw path, not the decoded path, for the URI and then returns a substring of that as the path. As a result the datanode ends up using a non-decoded path and we have a path mismatch between what the client requested and what the datanode tries to open on their behalf. Unable to open webhdfs paths with escape characters --- Key: HDFS-7816 URL: https://issues.apache.org/jira/browse/HDFS-7816 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.7.0 Reporter: Jason Lowe Priority: Blocker webhdfs requests to open files with % characters in the filename fail because the filename is not being decoded properly. For example: $ hadoop fs -cat 'webhdfs://nn/user/somebody/abc%def' cat: File does not exist: /user/somebody/abc%25def -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7279) Use netty to implement DatanodeWebHdfsMethods
[ https://issues.apache.org/jira/browse/HDFS-7279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329206#comment-14329206 ] Jason Lowe commented on HDFS-7279: -- We ran into an issue with webhdfs paths containing escape characters that seems to be related to this change. See HDFS-7816. Use netty to implement DatanodeWebHdfsMethods - Key: HDFS-7279 URL: https://issues.apache.org/jira/browse/HDFS-7279 Project: Hadoop HDFS Issue Type: Improvement Components: datanode, webhdfs Reporter: Haohui Mai Assignee: Haohui Mai Fix For: 2.7.0 Attachments: HDFS-7279.000.patch, HDFS-7279.001.patch, HDFS-7279.002.patch, HDFS-7279.003.patch, HDFS-7279.004.patch, HDFS-7279.005.patch, HDFS-7279.006.patch, HDFS-7279.007.patch, HDFS-7279.008.patch, HDFS-7279.009.patch, HDFS-7279.010.patch, HDFS-7279.011.patch, HDFS-7279.012.patch, HDFS-7279.013.patch Currently the DN implements all related webhdfs functionality using jetty. As the current jetty version the DN used (jetty 6) lacks of fine-grained buffer and connection management, DN often suffers from long latency and OOM when its webhdfs component is under sustained heavy load. This jira proposes to implement the webhdfs component in DN using netty, which can be more efficient and allow more finer-grain controls on webhdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7805) NameNode recovery prompt should be printed on console
[
https://issues.apache.org/jira/browse/HDFS-7805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329242#comment-14329242
]
Colin Patrick McCabe commented on HDFS-7805:
This should also be a System.out.println:
{code}
LOG.info(automatically choosing + firstChoice);
{code}
+1 once that's resolved. Thanks, surendra.
NameNode recovery prompt should be printed on console
-
Key: HDFS-7805
URL: https://issues.apache.org/jira/browse/HDFS-7805
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
Attachments: HDFS-7805.patch
In my cluster root logger is not console, so when I run namenode recovery
tool MetaRecoveryContext.java prompt message is logged in log file.
Actually is should be display on console.
Currently it is like this
{code}
LOG.info(prompt);
{code}
It should be
{code}
System.err.print(prompt);
{code}
NameNode recovery prompt should be printed on console
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7816) Unable to open webhdfs paths with escape characters
Jason Lowe created HDFS-7816: Summary: Unable to open webhdfs paths with escape characters Key: HDFS-7816 URL: https://issues.apache.org/jira/browse/HDFS-7816 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.7.0 Reporter: Jason Lowe Priority: Blocker webhdfs requests to open files with % characters in the filename fail because the filename is not being decoded properly. For example: $ hadoop fs -cat 'webhdfs://nn/user/somebody/abc%def' cat: File does not exist: /user/somebody/abc%25def -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7491) Add incremental blockreport latency to DN metrics
[ https://issues.apache.org/jira/browse/HDFS-7491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14330037#comment-14330037 ] Xiaoyu Yao commented on HDFS-7491: -- Hi Ming, could you use Time.monotonicNow() in elapsed time calculation? It will be more accurate and safe from system time changes. Otherwise, the patch looks good to me. Add incremental blockreport latency to DN metrics - Key: HDFS-7491 URL: https://issues.apache.org/jira/browse/HDFS-7491 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Reporter: Ming Ma Assignee: Ming Ma Priority: Minor Attachments: HDFS-7491-2.patch, HDFS-7491-3.patch, HDFS-7491-branch-2.patch, HDFS-7491.patch In a busy cluster, IBR processing could be delayed due to NN FSNamesystem lock and cause NN to throw NotReplicatedYetException to DFSClient and thus increase the overall application latency. This will be taken care of when we address the NN FSNamesystem lock contention issue. It is useful if we can provide IBR latency metrics from DN's point of view. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7816) Unable to open webhdfs paths with +
[ https://issues.apache.org/jira/browse/HDFS-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14330036#comment-14330036 ] Haohui Mai commented on HDFS-7816: -- bq. Patch looks good to me, +1 Reviews from non-committer are welcome, but it would be helpful for other people if you clarify this is a non-binding +1 to avoid confusion. Thanks. Unable to open webhdfs paths with + - Key: HDFS-7816 URL: https://issues.apache.org/jira/browse/HDFS-7816 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.7.0 Reporter: Jason Lowe Assignee: Kihwal Lee Priority: Blocker Attachments: HDFS-7816.patch, HDFS-7816.patch webhdfs requests to open files with % characters in the filename fail because the filename is not being decoded properly. For example: $ hadoop fs -cat 'webhdfs://nn/user/somebody/abc%def' cat: File does not exist: /user/somebody/abc%25def -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7740) Test truncate with DataNodes restarting
[ https://issues.apache.org/jira/browse/HDFS-7740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HDFS-7740: - Attachment: HDFS-7740.003.patch Test truncate with DataNodes restarting --- Key: HDFS-7740 URL: https://issues.apache.org/jira/browse/HDFS-7740 Project: Hadoop HDFS Issue Type: Sub-task Components: test Affects Versions: 2.7.0 Reporter: Konstantin Shvachko Assignee: Yi Liu Fix For: 2.7.0 Attachments: HDFS-7740.001.patch, HDFS-7740.002.patch, HDFS-7740.003.patch Add a test case, which ensures replica consistency when DNs are failing and restarting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7740) Test truncate with DataNodes restarting
[ https://issues.apache.org/jira/browse/HDFS-7740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HDFS-7740: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Update the patch for that change and commit it to trunk and branch-2. Thanks again [~shv], also thanks [~szetszwo]. Test truncate with DataNodes restarting --- Key: HDFS-7740 URL: https://issues.apache.org/jira/browse/HDFS-7740 Project: Hadoop HDFS Issue Type: Sub-task Components: test Affects Versions: 2.7.0 Reporter: Konstantin Shvachko Assignee: Yi Liu Fix For: 2.7.0 Attachments: HDFS-7740.001.patch, HDFS-7740.002.patch, HDFS-7740.003.patch Add a test case, which ensures replica consistency when DNs are failing and restarting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7009) Active NN and standby NN have different live nodes
[
https://issues.apache.org/jira/browse/HDFS-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14330061#comment-14330061
]
Hadoop QA commented on HDFS-7009:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/1270/HDFS-7009-4.patch
against trunk revision 6f01330.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9633//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9633//console
This message is automatically generated.
Active NN and standby NN have different live nodes
--
Key: HDFS-7009
URL: https://issues.apache.org/jira/browse/HDFS-7009
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Ming Ma
Assignee: Ming Ma
Attachments: HDFS-7009-2.patch, HDFS-7009-3.patch, HDFS-7009-4.patch,
HDFS-7009.patch
To follow up on https://issues.apache.org/jira/browse/HDFS-6478, in most
cases, given DN sends HB and BR to NN regularly, if a specific RPC call
fails, it isn't a big deal.
However, there are cases where DN fails to register with NN during initial
handshake due to exceptions not covered by RPC client's connection retry.
When this happens, the DN won't talk to that NN until the DN restarts.
{noformat}
BPServiceActor
public void run() {
LOG.info(this + starting to offer service);
try {
// init stuff
try {
// setup storage
connectToNNAndHandshake();
} catch (IOException ioe) {
// Initial handshake, storage recovery or registration failed
// End BPOfferService thread
LOG.fatal(Initialization failed for block pool + this, ioe);
return;
}
initialized = true; // bp is initialized;
while (shouldRun()) {
try {
offerService();
} catch (Exception ex) {
LOG.error(Exception in BPOfferService for + this, ex);
sleepAndLogInterrupts(5000, offering service);
}
}
...
{noformat}
Here is an example of the call stack.
{noformat}
java.io.IOException: Failed on local exception: java.io.IOException: Response
is null.; Host Details : local host is: xxx; destination host is:
yyy:8030;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:761)
at org.apache.hadoop.ipc.Client.call(Client.java:1239)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
at com.sun.proxy.$Proxy9.registerDatanode(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
at com.sun.proxy.$Proxy9.registerDatanode(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.registerDatanode(DatanodeProtocolClientSideTranslatorPB.java:146)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:623)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Response is null.
at
org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:949)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:844)
{noformat}
This will create discrepancy
[jira] [Commented] (HDFS-7740) Test truncate with DataNodes restarting
[ https://issues.apache.org/jira/browse/HDFS-7740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14330067#comment-14330067 ] Hudson commented on HDFS-7740: -- FAILURE: Integrated in Hadoop-trunk-Commit #7172 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7172/]) HDFS-7740. Test truncate with DataNodes restarting. (yliu) (yliu: rev 737bad02d4cf879fa7d20b7c0e083d9dc59f604c) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java Test truncate with DataNodes restarting --- Key: HDFS-7740 URL: https://issues.apache.org/jira/browse/HDFS-7740 Project: Hadoop HDFS Issue Type: Sub-task Components: test Affects Versions: 2.7.0 Reporter: Konstantin Shvachko Assignee: Yi Liu Fix For: 2.7.0 Attachments: HDFS-7740.001.patch, HDFS-7740.002.patch, HDFS-7740.003.patch Add a test case, which ensures replica consistency when DNs are failing and restarting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7814) Fix usage string of storageType parameter for dfsadmin -setSpaceQuota/clrSpaceQuota
[ https://issues.apache.org/jira/browse/HDFS-7814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329506#comment-14329506 ] Hudson commented on HDFS-7814: -- FAILURE: Integrated in Hadoop-trunk-Commit #7165 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7165/]) HDFS-7814. Fix usage string of storageType parameter for dfsadmin -setSpaceQuota/clrSpaceQuota. Contributed by Xiaoyu Yao. (cnauroth: rev 8c6ae0d6199efa327d8f2761f2ad2163a60e5508) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml Fix usage string of storageType parameter for dfsadmin -setSpaceQuota/clrSpaceQuota - Key: HDFS-7814 URL: https://issues.apache.org/jira/browse/HDFS-7814 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Priority: Minor Fix For: 2.7.0 Attachments: HDFS-7814.00.patch This was found when I'm documenting the quota by storage type feature. The current usage string to set/clear quota by storage type which put the -storageType prameter after dirnames is incorrect. hdfs dfsadmin -setSpaceQuota/clrSpaceQuota quota dirname...dirname -storageType storagetype. The correct one should be: hdfs dfsadmin -setSpaceQuota/clrSpaceQuota quota [-storageType storagetype] dirname...dirname I will post the fix shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7693) libhdfs: add hdfsFile cache
[ https://issues.apache.org/jira/browse/HDFS-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329593#comment-14329593 ] Chris Nauroth commented on HDFS-7693: - Thanks for taking the time to explain the motivation, Colin. It's still a potentially interesting idea, perhaps worth resurrecting at some point right in the Java client too. libhdfs: add hdfsFile cache --- Key: HDFS-7693 URL: https://issues.apache.org/jira/browse/HDFS-7693 Project: Hadoop HDFS Issue Type: Improvement Components: libhdfs Affects Versions: 2.7.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-7693.001.patch Add an hdfsFile cache inside libhdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7814) Fix usage string of storageType parameter for dfsadmin -setSpaceQuota/clrSpaceQuota
[ https://issues.apache.org/jira/browse/HDFS-7814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329632#comment-14329632 ] Xiaoyu Yao commented on HDFS-7814: -- Thanks [~cnauroth] for reviewing and committing the patch! Fix usage string of storageType parameter for dfsadmin -setSpaceQuota/clrSpaceQuota - Key: HDFS-7814 URL: https://issues.apache.org/jira/browse/HDFS-7814 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Priority: Minor Fix For: 2.7.0 Attachments: HDFS-7814.00.patch This was found when I'm documenting the quota by storage type feature. The current usage string to set/clear quota by storage type which put the -storageType prameter after dirnames is incorrect. hdfs dfsadmin -setSpaceQuota/clrSpaceQuota quota dirname...dirname -storageType storagetype. The correct one should be: hdfs dfsadmin -setSpaceQuota/clrSpaceQuota quota [-storageType storagetype] dirname...dirname I will post the fix shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7805) NameNode recovery prompt should be printed on console
[
https://issues.apache.org/jira/browse/HDFS-7805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14330034#comment-14330034
]
surendra singh lilhore commented on HDFS-7805:
--
[~cmccabe] Thanks :)
Attached new patch.
NameNode recovery prompt should be printed on console
-
Key: HDFS-7805
URL: https://issues.apache.org/jira/browse/HDFS-7805
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
Attachments: HDFS-7805.patch, HDFS-7805_1.patch
In my cluster root logger is not console, so when I run namenode recovery
tool MetaRecoveryContext.java prompt message is logged in log file.
Actually is should be display on console.
Currently it is like this
{code}
LOG.info(prompt);
{code}
It should be
{code}
System.err.print(prompt);
{code}
NameNode recovery prompt should be printed on console
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7816) Unable to open webhdfs paths with +
[ https://issues.apache.org/jira/browse/HDFS-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14330033#comment-14330033 ] Haohui Mai commented on HDFS-7816: -- bq. I don't think we can rely on clients changing the way the URL is encoded, otherwise we break compatibility with older clients. I think Kihwal's patch will work even with older clients. My main concern is that we're relying on QueryStringDecoder#path to give us a raw path so URI can decode it properly Speaking about compatibility, note that we need to consider the compatibility of other clients as well. For example, python clients expects WebHDFS server strictly follows URIs encoding scheme, that is, the URI that are sent over the wire strictly follows RFC 3986. This is well-defined. If the WebHDFS client happens to diverge from it, it should be considered as a bug instead of a feature that needs to be backward-compatible. Unable to open webhdfs paths with + - Key: HDFS-7816 URL: https://issues.apache.org/jira/browse/HDFS-7816 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.7.0 Reporter: Jason Lowe Assignee: Kihwal Lee Priority: Blocker Attachments: HDFS-7816.patch, HDFS-7816.patch webhdfs requests to open files with % characters in the filename fail because the filename is not being decoded properly. For example: $ hadoop fs -cat 'webhdfs://nn/user/somebody/abc%def' cat: File does not exist: /user/somebody/abc%25def -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7507) Fix findbugs warning in hadoop-hdfs project
[ https://issues.apache.org/jira/browse/HDFS-7507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDFS-7507. -- Resolution: Duplicate Fix findbugs warning in hadoop-hdfs project --- Key: HDFS-7507 URL: https://issues.apache.org/jira/browse/HDFS-7507 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.0 Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Find the warning when building patch for HDFS-7475. Full report: https://builds.apache.org/job/PreCommit-HDFS-Build/8993//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html#Warnings_BAD_PRACTICE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-5356) MiniDFSCluster shoud close all open FileSystems when shutdown()
[
https://issues.apache.org/jira/browse/HDFS-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329259#comment-14329259
]
Colin Patrick McCabe commented on HDFS-5356:
Hi Rakesh,
That's a very fair question. I agree that it's nicer in theory to have
{{MiniDFSCluster#shutdown}} call close on every FS.
What happened in practice is that almost all the long-lived resources have
moved out of FileSystem (and its delegate DFSClient) and into ClientContext.
This happened because the FileContext API doesn't have a close function, and a
lot of the code is shared.
FileSystem#close has also created a lot of problems over the years because
while we give out the same FS instance to multiple caller of FileSystem#get, we
don't reference count it (meaning that one thread which calls close on it can
shut it down even if there are other users). This shouldn't be a problem in
unit tests, but it is one reason hadoop FS closing doesn't happen that often in
real systems.
All that being said, I would be OK with the change you have now if you checked
whether the FS was in the fileSystems list prior to adding it. Currently it
appears like we may call close twice on the same fs if
MiniDFSCluster#getFileSystemInstance was called more than once.
MiniDFSCluster shoud close all open FileSystems when shutdown()
---
Key: HDFS-5356
URL: https://issues.apache.org/jira/browse/HDFS-5356
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Affects Versions: 3.0.0, 2.2.0
Reporter: haosdent
Assignee: Rakesh R
Priority: Critical
Attachments: HDFS-5356-1.patch, HDFS-5356-2.patch, HDFS-5356-3.patch,
HDFS-5356.patch
After add some metrics functions to DFSClient, I found that some unit tests
relates to metrics are failed. Because MiniDFSCluster are never close open
FileSystems, DFSClients are alive after MiniDFSCluster shutdown(). The
metrics of DFSClients in DefaultMetricsSystem are still exist and this make
other unit tests failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7360) Test libhdfs3 against MiniDFSCluster
[
https://issues.apache.org/jira/browse/HDFS-7360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329307#comment-14329307
]
Colin Patrick McCabe commented on HDFS-7360:
{code}
164 INCLUDE(CheckCSourceCompiles)
165 CHECK_C_SOURCE_COMPILES(
166 #include string.h
167
168 #ifdef _WIN32
169 #define strerror_r(errnum, buf, buflen) strerror_s((buf), (buflen),
(errnum))
170 #endif
171
172 int main(void)
173 {
174 int i = strerror_r(0, 0, 100);
175 return 0;
176 }
177 STRERROR_R_RETURN_INT)
{code}
This test isn't quite right. C/C++ can easily coerce a {{char*}} into an
{{int}} (or vice versa) and compile with an error. There will be a warning,
but STRERROR_R_RETURN_INT will still be set to true. Instead, you should do
the same thing we do with the {{terror}} function in libhdfs to make this work
(or even just copy that function).
Anyway, that is an existing problem, not a new one. The rest looks good. +1
Test libhdfs3 against MiniDFSCluster
Key: HDFS-7360
URL: https://issues.apache.org/jira/browse/HDFS-7360
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Reporter: Haohui Mai
Assignee: Zhanwei Wang
Priority: Critical
Attachments: HDFS-7360-pnative.002.patch, HDFS-7360.patch
Currently the branch has enough code to interact with HDFS servers. We should
test the code against MiniDFSCluster to ensure the correctness of the code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HDFS-6994) libhdfs3 - A native C/C++ HDFS client
[
https://issues.apache.org/jira/browse/HDFS-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329333#comment-14329333
]
Colin Patrick McCabe edited comment on HDFS-6994 at 2/20/15 6:47 PM:
-
bq. [~wheat9] wrote: I'm concerned about this. What are the guarantees of the
APIs for the releases? Are the APIs / ABIs going to be compatible once we
remove exceptions in later versions? Can the user simply do a drop-in
replacement to upgrade? For the part of libhdfs binding the answer might be
yes, but my general impression is no due to the complexity of SEH on Windows
and various quirks on the implementation of the C++ exceptions.
The current plan is to expose only the existing {{libhdfs.h}} API for now.
Since this is a C API, it does not include exceptions, clearly. So I do not
think there will be a problem with this.
What we are discussing is eliminating the use of exceptions internally. Since
this happens at a level which is not visible to users, it can certainly be done
later if we want to. However, I would like to see it fixed sooner rather than
later. it is important to stick to a consistent coding style, and we want to
work on the robustness.
I have proposed a C\+\+ API for libhdfs and libhdfs3 at
https://issues.apache.org/jira/browse/HDFS-7207. I would welcome more
discussion there. Note that my API does not use exceptions and does not
require C\+\+11 (although it can make use of C\+\+11 features if it is
available.)
bq. [asynchronous api discussion]
If you look at a high-performance HDFS client like Impala or HAWQ, they are
fine with synchronous APIs. Why? Well, most of the time your read performance
is limited by the bandwidth of the local disks (high performance clients always
try to do local reads, and use short-circuit and mmap if possible). A local
hard disk can't handle more than maybe 100 seeks a second, and the more seeks
you do, the lower your bandwidth will be.
There is also the CPU aspect: what are you doing with the data? Sure you can
have 10,000 async requests going with 1 thread, but if that thread is actually
doing anything with the data, you can cut a few zeroes off of that. And then
you're back to an amount of concurrent reads that can be comfortably done
synchronously.
Async APIs work best for cases where you are doing very, very little processing
on each request. So an async web server like ngnix, which is written in highly
optimized straight C (no \+\+) can squeeze a few more pages per second out of
reducing its thread count. But in a DB it's tougher (and as you mentioned, it
also makes the code much more complex).
So while we should probably consider an async client at some point, I think it
is much lower priority than other things (like finishing the existing native
client and merging it)
was (Author: cmccabe):
bq. [~wheat9] wrote: I'm concerned about this. What are the guarantees of the
APIs for the releases? Are the APIs / ABIs going to be compatible once we
remove exceptions in later versions? Can the user simply do a drop-in
replacement to upgrade? For the part of libhdfs binding the answer might be
yes, but my general impression is no due to the complexity of SEH on Windows
and various quirks on the implementation of the C++ exceptions.
The current plan is to expose only the existing {{libhdfs.h}} API for now.
Since this is a C API, it does not include exceptions, clearly. So I do not
think there will be a problem with this.
What we are discussing is eliminating the use of exceptions internally. Since
this happens at a level which is not visible to users, it can certainly be done
later if we want to. However, I would like to see it fixed sooner rather than
later. it is important to stick to a consistent coding style, and we want to
work on the robustness.
I have proposed a C\+\+ API for libhdfs and libhdfs3 at
https://issues.apache.org/jira/browse/HDFS-7207. I would welcome more
discussion there. Note that my API does not use exceptions and does not
require C\+\+11 (although it can make use of C\+\+ features if it is available.)
bq. [asynchronous api discussion]
If you look at a high-performance HDFS client like Impala or HAWQ, they are
fine with synchronous APIs. Why? Well, most of the time your read performance
is limited by the bandwidth of the local disks (high performance clients always
try to do local reads, and use short-circuit and mmap if possible). A local
hard disk can't handle more than maybe 100 seeks a second, and the more seeks
you do, the lower your bandwidth will be.
There is also the CPU aspect: what are you doing with the data? Sure you can
have 10,000 async requests going with 1 thread, but if that thread is actually
doing anything with the data, you can cut a few zeroes off of that. And then
you're back to an amount of concurrent reads that can
[jira] [Commented] (HDFS-7648) Verify the datanode directory layout
[ https://issues.apache.org/jira/browse/HDFS-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329289#comment-14329289 ] Colin Patrick McCabe commented on HDFS-7648: Latest patch looks better. Can we please not have the same code duplicated in two places, though? Use a function if needed, or just put it further up in the loop. bq. If we wait until it happened, how could we help the users? The may also be other reasons such as there is a bug in the software, the admin manually moved the block files around, etc. We can help the users by pointing out that there is a problem with the block layout. If there is a bug in the software, we cannot automatically fix the bug (I hope you will agree to this much, at least). And automatically undoing something that the admin did, even if that was a silly thing, is a bad thing to do. Verify the datanode directory layout Key: HDFS-7648 URL: https://issues.apache.org/jira/browse/HDFS-7648 Project: Hadoop HDFS Issue Type: Bug Components: datanode Reporter: Tsz Wo Nicholas Sze Assignee: Rakesh R Attachments: HDFS-7648-3.patch, HDFS-7648-4.patch, HDFS-7648-5.patch, HDFS-7648.patch, HDFS-7648.patch HDFS-6482 changed datanode layout to use block ID to determine the directory to store the block. We should have some mechanism to verify it. Either DirectoryScanner or block report generation could do the check. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6994) libhdfs3 - A native C/C++ HDFS client
[
https://issues.apache.org/jira/browse/HDFS-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329333#comment-14329333
]
Colin Patrick McCabe commented on HDFS-6994:
bq. [~wheat9] wrote: I'm concerned about this. What are the guarantees of the
APIs for the releases? Are the APIs / ABIs going to be compatible once we
remove exceptions in later versions? Can the user simply do a drop-in
replacement to upgrade? For the part of libhdfs binding the answer might be
yes, but my general impression is no due to the complexity of SEH on Windows
and various quirks on the implementation of the C++ exceptions.
The current plan is to expose only the existing {{libhdfs.h}} API for now.
Since this is a C API, it does not include exceptions, clearly. So I do not
think there will be a problem with this.
What we are discussing is eliminating the use of exceptions internally. Since
this happens at a level which is not visible to users, it can certainly be done
later if we want to. However, I would like to see it fixed sooner rather than
later. it is important to stick to a consistent coding style, and we want to
work on the robustness.
I have proposed a C\+\+ API for libhdfs and libhdfs3 at
https://issues.apache.org/jira/browse/HDFS-7207. I would welcome more
discussion there. Note that my API does not use exceptions and does not
require C\+\+11 (although it can make use of C\+\+ features if it is available.)
bq. [asynchronous api discussion]
If you look at a high-performance HDFS client like Impala or HAWQ, they are
fine with synchronous APIs. Why? Well, most of the time your read performance
is limited by the bandwidth of the local disks (high performance clients always
try to do local reads, and use short-circuit and mmap if possible). A local
hard disk can't handle more than maybe 100 seeks a second, and the more seeks
you do, the lower your bandwidth will be.
There is also the CPU aspect: what are you doing with the data? Sure you can
have 10,000 async requests going with 1 thread, but if that thread is actually
doing anything with the data, you can cut a few zeroes off of that. And then
you're back to an amount of concurrent reads that can be comfortably done
synchronously.
Async APIs work best for cases where you are doing very, very little processing
on each request. So an async web server like ngnix, which is written in highly
optimized straight C (no \+\+) can squeeze a few more pages per second out of
reducing its thread count. But in a DB it's tougher (and as you mentioned, it
also makes the code much more complex).
So while we should probably consider an async client at some point, I think it
is much lower priority than other things (like finishing the existing native
client and merging it)
libhdfs3 - A native C/C++ HDFS client
-
Key: HDFS-6994
URL: https://issues.apache.org/jira/browse/HDFS-6994
Project: Hadoop HDFS
Issue Type: New Feature
Components: hdfs-client
Reporter: Zhanwei Wang
Assignee: Zhanwei Wang
Attachments: HDFS-6994-rpc-8.patch, HDFS-6994.patch
Hi All
I just got the permission to open source libhdfs3, which is a native C/C++
HDFS client based on Hadoop RPC protocol and HDFS Data Transfer Protocol.
libhdfs3 provide the libhdfs style C interface and a C++ interface. Support
both HADOOP RPC version 8 and 9. Support Namenode HA and Kerberos
authentication.
libhdfs3 is currently used by HAWQ of Pivotal
I'd like to integrate libhdfs3 into HDFS source code to benefit others.
You can find libhdfs3 code from github
https://github.com/PivotalRD/libhdfs3
http://pivotalrd.github.io/libhdfs3/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7817) libhdfs3: fix strerror_r detection
Colin Patrick McCabe created HDFS-7817:
--
Summary: libhdfs3: fix strerror_r detection
Key: HDFS-7817
URL: https://issues.apache.org/jira/browse/HDFS-7817
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Colin Patrick McCabe
The signature of strerror_r is not quite detected correctly in libhdfs3. The
code assumes that {{int foo = strerror_r}} will fail to compile with the GNU
type signature, but this is not the case (C\+\+ will coerce the char* to an int
in this case). Instead, we should do what the libhdfs {{terror}} (threaded
error) function does here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7023) use libexpat instead of libxml2 for libhdfs3
[ https://issues.apache.org/jira/browse/HDFS-7023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-7023: --- Resolution: Fixed Status: Resolved (was: Patch Available) this was committed earlier use libexpat instead of libxml2 for libhdfs3 Key: HDFS-7023 URL: https://issues.apache.org/jira/browse/HDFS-7023 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Affects Versions: HDFS-6994 Reporter: Zhanwei Wang Assignee: Colin Patrick McCabe Fix For: HDFS-6994 Attachments: HDFS-7023-pnative.002.patch, HDFS-7023.001.pnative.patch As commented in HDFS-6994, libxml2 may has some thread safe issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7360) Test libhdfs3 against MiniDFSCluster
[ https://issues.apache.org/jira/browse/HDFS-7360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329351#comment-14329351 ] Colin Patrick McCabe commented on HDFS-7360: Filed HDFS-7817 for the strerror_r issue. Test libhdfs3 against MiniDFSCluster Key: HDFS-7360 URL: https://issues.apache.org/jira/browse/HDFS-7360 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Reporter: Haohui Mai Assignee: Zhanwei Wang Priority: Critical Attachments: HDFS-7360-pnative.002.patch, HDFS-7360.patch Currently the branch has enough code to interact with HDFS servers. We should test the code against MiniDFSCluster to ensure the correctness of the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HDFS-7816) Unable to open webhdfs paths with escape characters
[ https://issues.apache.org/jira/browse/HDFS-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula reassigned HDFS-7816: -- Assignee: Brahma Reddy Battula Unable to open webhdfs paths with escape characters --- Key: HDFS-7816 URL: https://issues.apache.org/jira/browse/HDFS-7816 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.7.0 Reporter: Jason Lowe Assignee: Brahma Reddy Battula Priority: Blocker webhdfs requests to open files with % characters in the filename fail because the filename is not being decoded properly. For example: $ hadoop fs -cat 'webhdfs://nn/user/somebody/abc%def' cat: File does not exist: /user/somebody/abc%25def -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7816) Unable to open webhdfs paths with escape characters
[ https://issues.apache.org/jira/browse/HDFS-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai resolved HDFS-7816. -- Resolution: Duplicate Fixed in HDFS-6662. Unable to open webhdfs paths with escape characters --- Key: HDFS-7816 URL: https://issues.apache.org/jira/browse/HDFS-7816 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.7.0 Reporter: Jason Lowe Assignee: Brahma Reddy Battula Priority: Blocker webhdfs requests to open files with % characters in the filename fail because the filename is not being decoded properly. For example: $ hadoop fs -cat 'webhdfs://nn/user/somebody/abc%def' cat: File does not exist: /user/somebody/abc%25def -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7816) Unable to open webhdfs paths with escape characters
[ https://issues.apache.org/jira/browse/HDFS-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329396#comment-14329396 ] Jason Lowe commented on HDFS-7816: -- Thanks, Haohui! Sorry for the duplicate noise, I missed HDFS-6662 when filing this. Unable to open webhdfs paths with escape characters --- Key: HDFS-7816 URL: https://issues.apache.org/jira/browse/HDFS-7816 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.7.0 Reporter: Jason Lowe Assignee: Brahma Reddy Battula Priority: Blocker webhdfs requests to open files with % characters in the filename fail because the filename is not being decoded properly. For example: $ hadoop fs -cat 'webhdfs://nn/user/somebody/abc%def' cat: File does not exist: /user/somebody/abc%25def -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7813) TestDFSHAAdminMiniCluster#testFencer testcase is failing frequently
[
https://issues.apache.org/jira/browse/HDFS-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-7813:
Resolution: Fixed
Fix Version/s: 2.7.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
+1 for the patch. I committed it to trunk and branch-2. Rakesh, thank you for
investigating the test failure and contributing a fix.
TestDFSHAAdminMiniCluster#testFencer testcase is failing frequently
---
Key: HDFS-7813
URL: https://issues.apache.org/jira/browse/HDFS-7813
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, test
Reporter: Rakesh R
Assignee: Rakesh R
Fix For: 2.7.0
Attachments: HDFS-7813-001.patch
Following is the failure trace.
{code}
java.lang.AssertionError: expected:0 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster.testFencer(TestDFSHAAdminMiniCluster.java:163)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7816) Unable to open webhdfs paths with +
[ https://issues.apache.org/jira/browse/HDFS-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329931#comment-14329931 ] Brahma Reddy Battula commented on HDFS-7816: It was broken from HDFS-6662... Patch looks good to me, +1..Hope you manually also verified in webUI... (I thought of give patch,by the time I woke up it was in patch available state :( ) Unable to open webhdfs paths with + - Key: HDFS-7816 URL: https://issues.apache.org/jira/browse/HDFS-7816 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.7.0 Reporter: Jason Lowe Assignee: Kihwal Lee Priority: Blocker Attachments: HDFS-7816.patch, HDFS-7816.patch webhdfs requests to open files with % characters in the filename fail because the filename is not being decoded properly. For example: $ hadoop fs -cat 'webhdfs://nn/user/somebody/abc%def' cat: File does not exist: /user/somebody/abc%25def -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7773) Additional metrics in HDFS to be accessed via jmx.
[ https://issues.apache.org/jira/browse/HDFS-7773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-7773: --- Attachment: hdfs-7773.branch-2.001.patch Additional metrics in HDFS to be accessed via jmx. -- Key: HDFS-7773 URL: https://issues.apache.org/jira/browse/HDFS-7773 Project: Hadoop HDFS Issue Type: Improvement Components: datanode, namenode Reporter: Anu Engineer Assignee: Anu Engineer Attachments: hdfs-7773.001.patch, hdfs-7773.002.patch, hdfs-7773.003.patch, hdfs-7773.branch-2.001.patch We would like to have the following metrics added to DataNode and name node this to improve Ambari dashboard 1) DN disk i/o utilization 2) DN network i/o utilization 3) Namenode read operations 4) Namenode write operations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7773) Additional metrics in HDFS to be accessed via jmx.
[ https://issues.apache.org/jira/browse/HDFS-7773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-7773: --- Status: Open (was: Patch Available) Creating a new patch for branch 2 too Additional metrics in HDFS to be accessed via jmx. -- Key: HDFS-7773 URL: https://issues.apache.org/jira/browse/HDFS-7773 Project: Hadoop HDFS Issue Type: Improvement Components: datanode, namenode Reporter: Anu Engineer Assignee: Anu Engineer Attachments: hdfs-7773.001.patch, hdfs-7773.002.patch, hdfs-7773.003.patch, hdfs-7773.branch-2.001.patch We would like to have the following metrics added to DataNode and name node this to improve Ambari dashboard 1) DN disk i/o utilization 2) DN network i/o utilization 3) Namenode read operations 4) Namenode write operations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HDFS-7816) Unable to open webhdfs paths with escape characters
[
https://issues.apache.org/jira/browse/HDFS-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329483#comment-14329483
]
Kihwal Lee edited comment on HDFS-7816 at 2/20/15 8:17 PM:
---
HDFS-6662 only partially fixes the problem. The decoding is done by calling
{{decodeComponent()}}, which decodes \+ as a space. A \+ sign in query
string should be decoded as a space, but not if in the path.
For proper decoding of path, we can create a {{URI}} object with the encoded
string and then call getPath(), which only decodes %, not \+.
was (Author: kihwal):
HDFS-6662 only partially fixes the problem. The decoding is done by calling
{{decodeComponent()}}, which decodes + as a space. A + sign in query string
should be decoded as a space, but not if in the path.
For proper decoding of path, we can create a {{URI}} object with the encoded
string and then call getPath(), which only decodes %, not +.
Unable to open webhdfs paths with escape characters
---
Key: HDFS-7816
URL: https://issues.apache.org/jira/browse/HDFS-7816
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.7.0
Reporter: Jason Lowe
Priority: Blocker
webhdfs requests to open files with % characters in the filename fail because
the filename is not being decoded properly. For example:
$ hadoop fs -cat 'webhdfs://nn/user/somebody/abc%def'
cat: File does not exist: /user/somebody/abc%25def
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HDFS-7816) Unable to open webhdfs paths with escape characters
[
https://issues.apache.org/jira/browse/HDFS-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee reopened HDFS-7816:
--
Assignee: (was: Brahma Reddy Battula)
HDFS-6662 only partially fixes the problem. The decoding is done by calling
{{decodeComponent()}}, which decodes + as a space. A + sign in query string
should be decoded as a space, but not if in the path.
For proper decoding of path, we can create a {{URI}} object with the encoded
string and then call getPath(), which only decodes %, not +.
Unable to open webhdfs paths with escape characters
---
Key: HDFS-7816
URL: https://issues.apache.org/jira/browse/HDFS-7816
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.7.0
Reporter: Jason Lowe
Priority: Blocker
webhdfs requests to open files with % characters in the filename fail because
the filename is not being decoded properly. For example:
$ hadoop fs -cat 'webhdfs://nn/user/somebody/abc%def'
cat: File does not exist: /user/somebody/abc%25def
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
