[jira] [Updated] (HDFS-8215) Refactor NamenodeFsck#check method
[ https://issues.apache.org/jira/browse/HDFS-8215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-8215: --- Status: Patch Available (was: Open) Refactor NamenodeFsck#check method -- Key: HDFS-8215 URL: https://issues.apache.org/jira/browse/HDFS-8215 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Takanobu Asanuma Assignee: Takanobu Asanuma Attachments: HDFS-8215.1.patch This method is very long, approximately 230 lines. Splitting this method into smaller methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6763) Perform file system-wide quota initialization once on transitioning to active
[
https://issues.apache.org/jira/browse/HDFS-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-6763:
-
Target Version/s: 2.8.0 (was: 2.6.0)
Perform file system-wide quota initialization once on transitioning to active
-
Key: HDFS-6763
URL: https://issues.apache.org/jira/browse/HDFS-6763
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, namenode
Reporter: Daryn Sharp
Attachments: HDFS-6763.patch
{{FSImage#loadEdits}} calls {{updateCountForQuota}} to recalculate verify
quotas for the entire namespace. A standby NN using shared edits calls this
method every minute. The standby may appear to hang for many seconds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8211) DataNode UUID is always null in the JMX counter
[
https://issues.apache.org/jira/browse/HDFS-8211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509357#comment-14509357
]
Anu Engineer commented on HDFS-8211:
I will make the test simpler, thanx for the pointers
bq. The test case seems to pass without the code change.
The reason being that all the functions of DataNode are private and it is not
possible to trigger the error path from outside. The checkuuid call would write
the UUID to storage variable and then on the connection complete it gets copied
to id. so the fix is just to read from the storage variable rather than id
since both f them point to same values.
At this point the test simply verifies that new code change that is made does
not lead to regressions, it does *not* test for the initial race condition
DataNode UUID is always null in the JMX counter
---
Key: HDFS-8211
URL: https://issues.apache.org/jira/browse/HDFS-8211
Project: Hadoop HDFS
Issue Type: Bug
Components: HDFS
Affects Versions: 2.7.0
Reporter: Anu Engineer
Assignee: Anu Engineer
Priority: Minor
Attachments: hdfs-8211.001.patch
The DataNode JMX counters are tagged with DataNode UUID, but it always gets a
null value instead of the UUID.
{code}
Hadoop:service=DataNode,name=FSDatasetState*-null*.
{code}
This null is supposed be the datanode UUID.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8211) DataNode UUID is always null in the JMX counter
[
https://issues.apache.org/jira/browse/HDFS-8211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509402#comment-14509402
]
Arpit Agarwal commented on HDFS-8211:
-
The storageUuid/storageId is different from the DatanodeUuid. The former is
allocated per storage directory. Yes it is confusing. :-)
I suggested using SimulatedFSDataSet so we can bypass this code block in
initStorage:
{code}
if (!factory.isSimulated()) {
final StartupOption startOpt = getStartupOption(conf);
...
{code}
Then all we'd need is a spy object for DataStorage which does nothing when
writeAll is invoked.
DataNode UUID is always null in the JMX counter
---
Key: HDFS-8211
URL: https://issues.apache.org/jira/browse/HDFS-8211
Project: Hadoop HDFS
Issue Type: Bug
Components: HDFS
Affects Versions: 2.7.0
Reporter: Anu Engineer
Assignee: Anu Engineer
Priority: Minor
Attachments: hdfs-8211.001.patch
The DataNode JMX counters are tagged with DataNode UUID, but it always gets a
null value instead of the UUID.
{code}
Hadoop:service=DataNode,name=FSDatasetState*-null*.
{code}
This null is supposed be the datanode UUID.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8232) Missing datanode counters when using Metrics2 sink interface
[ https://issues.apache.org/jira/browse/HDFS-8232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-8232: --- Description: When using the Metric2 Sink interface none of the counters declared under Dataanode:FSDataSetBean are visible. They are visible if you use JMX or if you do http://host:port/jmx. Expected behavior is that they be part of Sink interface and accessible in the putMetrics call back. was: When using the Metric2 Sink interface none of the counters declared under Dataanode:FSDataSetBean are not visible. They are visible if you use JMX or if you do http://host:port/jmx. Expected behavior is that they be part of Sink interface and accessible in the putMetrics call back. Missing datanode counters when using Metrics2 sink interface Key: HDFS-8232 URL: https://issues.apache.org/jira/browse/HDFS-8232 Project: Hadoop HDFS Issue Type: Bug Components: HDFS Affects Versions: 2.7.1 Reporter: Anu Engineer Assignee: Anu Engineer When using the Metric2 Sink interface none of the counters declared under Dataanode:FSDataSetBean are visible. They are visible if you use JMX or if you do http://host:port/jmx. Expected behavior is that they be part of Sink interface and accessible in the putMetrics call back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8232) Missing datanode counters when using Metrics2 sink interface
Anu Engineer created HDFS-8232: -- Summary: Missing datanode counters when using Metrics2 sink interface Key: HDFS-8232 URL: https://issues.apache.org/jira/browse/HDFS-8232 Project: Hadoop HDFS Issue Type: Bug Components: HDFS Affects Versions: 2.7.1 Reporter: Anu Engineer Assignee: Anu Engineer When using the Metric2 Sink interface none of the counters declared under Dataanode:FSDataSetBean are not visible. They are visible if you use JMX or if you do http://host:port/jmx. Expected behavior is that they be part of Sink interface and accessible in the putMetrics call back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8204) Mover/Balancer should not schedule two replicas to the same DN
[
https://issues.apache.org/jira/browse/HDFS-8204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509554#comment-14509554
]
Tsz Wo Nicholas Sze commented on HDFS-8204:
---
My previous comment was for the bug, not the patch.
What's more important, 2 replicas located on 1 DN doesn't match placement
policy.
I am confused. Do you actually see 2 replicas stored in 1 DN? Or you only see
two replicas *scheduled* to be moved to the same DN?
Mover/Balancer should not schedule two replicas to the same DN
--
Key: HDFS-8204
URL: https://issues.apache.org/jira/browse/HDFS-8204
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer mover
Reporter: Walter Su
Assignee: Walter Su
Attachments: HDFS-8204.001.patch
Balancer moves blocks between Datanode(Ver. 2.6 ).
Balancer moves blocks between StorageGroups ( introduced by HDFS-6584) , in
the new version(Ver. =2.6) .
function
{code}
class DBlock extends LocationsStorageGroup
DBlock.isLocatedOn(StorageGroup loc)
{code}
is flawed, may causes 2 replicas ends in same node after running balance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8213) DFSClient should use hdfs.client.htrace HTrace configuration prefix rather than hadoop.htrace
[ https://issues.apache.org/jira/browse/HDFS-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-8213: --- Summary: DFSClient should use hdfs.client.htrace HTrace configuration prefix rather than hadoop.htrace (was: DFSClient should not instantiate SpanReceiverHost) DFSClient should use hdfs.client.htrace HTrace configuration prefix rather than hadoop.htrace - Key: HDFS-8213 URL: https://issues.apache.org/jira/browse/HDFS-8213 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.0 Reporter: Billie Rinaldi Assignee: Brahma Reddy Battula Priority: Critical DFSClient initializing SpanReceivers is a problem for Accumulo, which manages SpanReceivers through its own configuration. This results in the same receivers being registered multiple times and spans being delivered more than once. The documentation says SpanReceiverHost.getInstance should be issued once per process, so there is no expectation that DFSClient should do this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8231) StackTrace displayed at client while QuotaByStorageType exceeds
[
https://issues.apache.org/jira/browse/HDFS-8231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509383#comment-14509383
]
Chris Nauroth commented on HDFS-8231:
-
[~andreina], thank you for reporting the bug. The fix is likely to track down
the various points in {{DFSClient}} that call
{{RemoteException#unwrapRemoteException}} on {{DSQuotaExceededException.class}}
and add {{QuotaByStorageTypeExceededException.class}} to the list.
StackTrace displayed at client while QuotaByStorageType exceeds
---
Key: HDFS-8231
URL: https://issues.apache.org/jira/browse/HDFS-8231
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Reporter: J.Andreina
Assignee: J.Andreina
Priority: Minor
StackTrace displayed at client while QuotaByStorageType exceeds.
With reference to HDFS-2360, feel better to fix this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8204) Mover/Balancer should not schedule two replicas to the same DN
[
https://issues.apache.org/jira/browse/HDFS-8204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509560#comment-14509560
]
Tsz Wo Nicholas Sze commented on HDFS-8204:
---
Just found that you updated the summary. So the bug is only in scheduling but
it does not decease the number of replicas. Then, it is the same as HDFS-8147.
Mover/Balancer should not schedule two replicas to the same DN
--
Key: HDFS-8204
URL: https://issues.apache.org/jira/browse/HDFS-8204
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer mover
Reporter: Walter Su
Assignee: Walter Su
Attachments: HDFS-8204.001.patch
Balancer moves blocks between Datanode(Ver. 2.6 ).
Balancer moves blocks between StorageGroups ( introduced by HDFS-6584) , in
the new version(Ver. =2.6) .
function
{code}
class DBlock extends LocationsStorageGroup
DBlock.isLocatedOn(StorageGroup loc)
{code}
is flawed, may causes 2 replicas ends in same node after running balance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8211) DataNode UUID is always null in the JMX counter
[
https://issues.apache.org/jira/browse/HDFS-8211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509377#comment-14509377
]
Anu Engineer commented on HDFS-8211:
I looked at the SimulatedFSDataSet unfortunately it always have been doing the
right thing, it is only now that I noticed it though :)
{code:title=SimulatedFSDataSet.java}
@Override
public String getStorageUuid() {
return storage.getStorageUuid();
}
{code}
So invoking it will pass my tests, but I am afraid would not touch the code
that path I modified right now. Please let me know if my understanding is
correct
DataNode UUID is always null in the JMX counter
---
Key: HDFS-8211
URL: https://issues.apache.org/jira/browse/HDFS-8211
Project: Hadoop HDFS
Issue Type: Bug
Components: HDFS
Affects Versions: 2.7.0
Reporter: Anu Engineer
Assignee: Anu Engineer
Priority: Minor
Attachments: hdfs-8211.001.patch
The DataNode JMX counters are tagged with DataNode UUID, but it always gets a
null value instead of the UUID.
{code}
Hadoop:service=DataNode,name=FSDatasetState*-null*.
{code}
This null is supposed be the datanode UUID.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6763) Perform file system-wide quota initialization once on transitioning to active
[
https://issues.apache.org/jira/browse/HDFS-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-6763:
-
Summary: Perform file system-wide quota initialization once on
transitioning to active (was: Standby NN stalls after processing edits)
Perform file system-wide quota initialization once on transitioning to active
-
Key: HDFS-6763
URL: https://issues.apache.org/jira/browse/HDFS-6763
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, namenode
Reporter: Daryn Sharp
{{FSImage#loadEdits}} calls {{updateCountForQuota}} to recalculate verify
quotas for the entire namespace. A standby NN using shared edits calls this
method every minute. The standby may appear to hang for many seconds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[
https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509407#comment-14509407
]
Hadoop QA commented on HDFS-8033:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 42s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | javac | 7m 38s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 49s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 3m 57s | There were no new checkstyle
issues. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 5s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | native | 3m 13s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 184m 28s | Tests passed in hadoop-hdfs.
|
| | | 229m 31s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12727610/ByteBufferStrategy_support_len.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / baf8bc6 |
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/10359/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/10359/testReport/ |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10359/console |
This message was automatically generated.
Erasure coding: stateful (non-positional) read from files in striped layout
---
Key: HDFS-8033
URL: https://issues.apache.org/jira/browse/HDFS-8033
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: ByteBufferStrategy_support_len.patch,
HDFS-8033.000.patch, HDFS-8033.001.patch, HDFS-8033.002.patch,
HDFS-8033.003.patch, hdfs8033-HDFS-7285.04.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HDFS-6763) Perform file system-wide quota initialization once on transitioning to active
[
https://issues.apache.org/jira/browse/HDFS-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee reassigned HDFS-6763:
Assignee: Kihwal Lee
Perform file system-wide quota initialization once on transitioning to active
-
Key: HDFS-6763
URL: https://issues.apache.org/jira/browse/HDFS-6763
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, namenode
Reporter: Daryn Sharp
Assignee: Kihwal Lee
Attachments: HDFS-6763.patch
{{FSImage#loadEdits}} calls {{updateCountForQuota}} to recalculate verify
quotas for the entire namespace. A standby NN using shared edits calls this
method every minute. The standby may appear to hang for many seconds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6763) Perform file system-wide quota initialization once on transitioning to active
[
https://issues.apache.org/jira/browse/HDFS-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-6763:
-
Attachment: HDFS-6763.patch
Perform file system-wide quota initialization once on transitioning to active
-
Key: HDFS-6763
URL: https://issues.apache.org/jira/browse/HDFS-6763
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, namenode
Reporter: Daryn Sharp
Assignee: Kihwal Lee
Attachments: HDFS-6763.patch
{{FSImage#loadEdits}} calls {{updateCountForQuota}} to recalculate verify
quotas for the entire namespace. A standby NN using shared edits calls this
method every minute. The standby may appear to hang for many seconds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6763) Standby NN stalls after processing edits
[
https://issues.apache.org/jira/browse/HDFS-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509389#comment-14509389
]
Kihwal Lee commented on HDFS-6763:
--
It is not just SBN that is being affected. Even ANN will traverse the entire
tree and update the quota each time it loads an edit segment on start-up. The
only place it is needed is when a NN is transitioning to active. All other use
cases of {{loadEdits()}} do not need quota to be updated:
{{BackupImage}}/{{BackupNode}}, {{Checkpointer}}, {{EditLogTailer}}. Quota is
used only when serving. We can simply make {{FSNamesystem}} do it once
whenever it transitions to active.
Standby NN stalls after processing edits
Key: HDFS-6763
URL: https://issues.apache.org/jira/browse/HDFS-6763
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, namenode
Reporter: Daryn Sharp
{{FSImage#loadEdits}} calls {{updateCountForQuota}} to recalculate verify
quotas for the entire namespace. A standby NN using shared edits calls this
method every minute. The standby may appear to hang for many seconds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8147) Mover should not schedule two replicas to the same DN storage
[
https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-8147:
--
Hadoop Flags: Reviewed
+1 the new patch looks good.
Mover should not schedule two replicas to the same DN storage
-
Key: HDFS-8147
URL: https://issues.apache.org/jira/browse/HDFS-8147
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer mover
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch,
HDFS-8147_3.patch, HDFS-8147_4.patch
*Scenario:*
1. Three DN cluster. For DNs storage type is like this.
DN1 : DISK,ARCHIVE
DN2 : DISK
DN3 : DISK,ARCHIVE (All DNs are in same rack)
2. One file with two replicas (In DN1 and DN2)
3. Set file storage policy COLD
4. Now execute Mover.
*Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE
*Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to
D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is
selecting again DN1:ARCHIVE for target where already same block exists.
{{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the
node as target where already same replica exists. The dispatcher will fail
to move block as shown in the log below. Then, the Mover will try again in
next iteration.
*Logs*
{code}
15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move
blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to
10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status
message opReplaceBlock
BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received
exception
org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica
FinalizedReplica, blk_1073741852_1028, FINALIZED
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8215) Refactor NamenodeFsck#check method
[
https://issues.apache.org/jira/browse/HDFS-8215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509704#comment-14509704
]
Hadoop QA commented on HDFS-8215:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 33s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | javac | 7m 27s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 38s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 6m 55s | The applied patch generated 1
additional checkstyle issues. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 6s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | native | 3m 14s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 175m 16s | Tests failed in hadoop-hdfs. |
| | | 222m 44s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestDFSClientRetries |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12727591/HDFS-8215.1.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 189a63a |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/10360/artifact/patchprocess/checkstyle-result-diff.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/10360/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/10360/testReport/ |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10360/console |
This message was automatically generated.
Refactor NamenodeFsck#check method
--
Key: HDFS-8215
URL: https://issues.apache.org/jira/browse/HDFS-8215
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Takanobu Asanuma
Assignee: Takanobu Asanuma
Attachments: HDFS-8215.1.patch
This method is very long, approximately 230 lines. Splitting this method into
smaller methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8204) Mover/Balancer should not schedule two replicas to the same DN
[
https://issues.apache.org/jira/browse/HDFS-8204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509608#comment-14509608
]
Tsz Wo Nicholas Sze commented on HDFS-8204:
---
Update: This issue is about DN. HDFS-8147 is about Storage. It's
different.
I see. Then, please checking the same storage group is not good enough. It
should check also the node.
Some comments for the patch:
- The logic could be simplified as below.
{code}
final DatanodeInfo targetDatanode = target.getDatanodeInfo();
for (StorageGroup blockLocation : block.getLocations()) {
if (blockLocation.getDatanodeInfo().equals(targetDatanode)) {
return false;
}
}
{code}
- The following is not needed since isGoodBlockCandidate is already called via
source.addPendingMove(db, target).
{code}
+if (!dispatcher.isGoodBlockCandidate(source, target, t, db)) {
+ continue;
+}
{code}
- Could you add/modify a test?
Mover/Balancer should not schedule two replicas to the same DN
--
Key: HDFS-8204
URL: https://issues.apache.org/jira/browse/HDFS-8204
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer mover
Reporter: Walter Su
Assignee: Walter Su
Attachments: HDFS-8204.001.patch
Balancer moves blocks between Datanode(Ver. 2.6 ).
Balancer moves blocks between StorageGroups ( introduced by HDFS-6584) , in
the new version(Ver. =2.6) .
function
{code}
class DBlock extends LocationsStorageGroup
DBlock.isLocatedOn(StorageGroup loc)
{code}
is flawed, may causes 2 replicas ends in same node after running balance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8233) Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when the last stripe is at the block group boundary
Jing Zhao created HDFS-8233:
---
Summary: Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when
the last stripe is at the block group boundary
Key: HDFS-8233
URL: https://issues.apache.org/jira/browse/HDFS-8233
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Jing Zhao
Assignee: Jing Zhao
Currently {{DFSStripedOutputStream#getCurrentBlockGroupBytes}} simply uses
{{getBytesCurBlock}} of each streamer to calculate the block group size. This
is wrong when the last stripe is at the block group boundary, since the
{{bytesCurBlock}} is set to 0 if an internal block is finished.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7337) Configurable and pluggable Erasure Codec and schema
[ https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509711#comment-14509711 ] Rashmi Vinayak commented on HDFS-7337: -- Hi Kai, For each of the modes in Hitchhiker, configuration would be identical to RS (as [~jack_liuquan] also mentioned). So we could have schemas like HH-k10-m4 just as in RS. Configurable and pluggable Erasure Codec and schema --- Key: HDFS-7337 URL: https://issues.apache.org/jira/browse/HDFS-7337 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Kai Zheng Attachments: HDFS-7337-prototype-v1.patch, HDFS-7337-prototype-v2.zip, HDFS-7337-prototype-v3.zip, PluggableErasureCodec-v2.pdf, PluggableErasureCodec-v3.pdf, PluggableErasureCodec.pdf According to HDFS-7285 and the design, this considers to support multiple Erasure Codecs via pluggable approach. It allows to define and configure multiple codec schemas with different coding algorithms and parameters. The resultant codec schemas can be utilized and specified via command tool for different file folders. While design and implement such pluggable framework, it’s also to implement a concrete codec by default (Reed Solomon) to prove the framework is useful and workable. Separate JIRA could be opened for the RS codec implementation. Note HDFS-7353 will focus on the very low level codec API and implementation to make concrete vendor libraries transparent to the upper layer. This JIRA focuses on high level stuffs that interact with configuration, schema and etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8147) Mover should not schedule two replicas to the same DN storage
[
https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509599#comment-14509599
]
Hudson commented on HDFS-8147:
--
FAILURE: Integrated in Hadoop-trunk-Commit #7652 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7652/])
HDFS-8147. StorageGroup in Dispatcher should override equals nad hashCode.
Contributed by surendra singh lilhore (szetszwo: rev
416b84354ebcc6f6f23598a5a254e62eb606680d)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
Mover should not schedule two replicas to the same DN storage
-
Key: HDFS-8147
URL: https://issues.apache.org/jira/browse/HDFS-8147
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer mover
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
Fix For: 2.7.1
Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch,
HDFS-8147_3.patch, HDFS-8147_4.patch
*Scenario:*
1. Three DN cluster. For DNs storage type is like this.
DN1 : DISK,ARCHIVE
DN2 : DISK
DN3 : DISK,ARCHIVE (All DNs are in same rack)
2. One file with two replicas (In DN1 and DN2)
3. Set file storage policy COLD
4. Now execute Mover.
*Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE
*Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to
D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is
selecting again DN1:ARCHIVE for target where already same block exists.
{{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the
node as target where already same replica exists. The dispatcher will fail
to move block as shown in the log below. Then, the Mover will try again in
next iteration.
*Logs*
{code}
15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move
blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to
10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status
message opReplaceBlock
BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received
exception
org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica
FinalizedReplica, blk_1073741852_1028, FINALIZED
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8234) DistributedFileSystem and Globber should apply PathFilter early
[
https://issues.apache.org/jira/browse/HDFS-8234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohini Palaniswamy updated HDFS-8234:
-
Description:
HDFS-985 added partial listing in listStatus to avoid listing entries of large
directory in one go. If listStatus(Path p, PathFilter f) call is made, filter
is applied after fetching all the entries resulting in a big list being
constructed on the client side. If the
DistributedFileSystem.listStatusInternal() applied the PathFilter it would be
more efficient. So DistributedFileSystem should override listStatus(Path f,
PathFilter filter) and apply PathFilter early.
Globber.java also applies filter after calling listStatus. It should call
listStatus with the PathFilter.
{code}
FileStatus[] children = listStatus(candidate.getPath());
.
for (FileStatus child : children) {
// Set the child path based on the parent path.
child.setPath(new Path(candidate.getPath(),
child.getPath().getName()));
if (globFilter.accept(child.getPath())) {
newCandidates.add(child);
}
}
{code}
was:HDFS-985 added partial listing in listStatus to avoid listing entries of
large directory in one go. If listStatus(Path p, PathFilter f) call is made,
filter is applied after fetching all the entries resulting in a big list being
constructed on the client side. If the
DistributedFileSystem.listStatusInternal() applied the PathFilter it would be
more efficient.
Summary: DistributedFileSystem and Globber should apply PathFilter
early (was: DistributedFileSystem should override listStatus(Path f,
PathFilter filter) )
DistributedFileSystem and Globber should apply PathFilter early
---
Key: HDFS-8234
URL: https://issues.apache.org/jira/browse/HDFS-8234
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Rohini Palaniswamy
Labels: newbie
HDFS-985 added partial listing in listStatus to avoid listing entries of
large directory in one go. If listStatus(Path p, PathFilter f) call is made,
filter is applied after fetching all the entries resulting in a big list
being constructed on the client side. If the
DistributedFileSystem.listStatusInternal() applied the PathFilter it would be
more efficient. So DistributedFileSystem should override listStatus(Path f,
PathFilter filter) and apply PathFilter early.
Globber.java also applies filter after calling listStatus. It should call
listStatus with the PathFilter.
{code}
FileStatus[] children = listStatus(candidate.getPath());
.
for (FileStatus child : children) {
// Set the child path based on the parent path.
child.setPath(new Path(candidate.getPath(),
child.getPath().getName()));
if (globFilter.accept(child.getPath())) {
newCandidates.add(child);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7959) WebHdfs logging is missing on Datanode
[ https://issues.apache.org/jira/browse/HDFS-7959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509956#comment-14509956 ] Haohui Mai commented on HDFS-7959: -- The current approach looks okay. A cleaner approach might be to introduce a LoggingHandler which catches all the response in downstream of the netty pipeline. WebHdfs logging is missing on Datanode -- Key: HDFS-7959 URL: https://issues.apache.org/jira/browse/HDFS-7959 Project: Hadoop HDFS Issue Type: Bug Reporter: Kihwal Lee Assignee: Kihwal Lee Priority: Critical Attachments: HDFS-7959.patch, HDFS-7959.patch, HDFS-7959.patch After the conversion to netty, webhdfs requests are not logged on datanodes. The existing jetty log only logs the non-webhdfs requests that come through the internal proxy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8235) Create DFSStripedInputStream in DFSClient#open
Jing Zhao created HDFS-8235: --- Summary: Create DFSStripedInputStream in DFSClient#open Key: HDFS-8235 URL: https://issues.apache.org/jira/browse/HDFS-8235 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Jing Zhao Currently DFSClient#open can only create a DFSInputStream object. It should also support DFSStripedInputStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8223) Should calculate checksum for parity blocks in DFSStripedOutputStream
[
https://issues.apache.org/jira/browse/HDFS-8223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509985#comment-14509985
]
Jing Zhao commented on HDFS-8223:
-
Thanks for fixing this, Yi! The patch looks good to me. +1
Besides, the patch needs rebase. You can do it while committing.
Should calculate checksum for parity blocks in DFSStripedOutputStream
-
Key: HDFS-8223
URL: https://issues.apache.org/jira/browse/HDFS-8223
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-8223.001.patch
{quote}
DFSPacket p = createPacket(packetSize, chunksPerPacket,
streamer.getBytesCurBlock(),
streamer.getAndIncCurrentSeqno(), false);
int maxBytesToPacket = p.getMaxChunks() * bytesPerChecksum;
int toWrite = byteBuffer.remaining() maxBytesToPacket ?
maxBytesToPacket: byteBuffer.remaining();
p.writeData(byteBuffer, toWrite);
streamer.incBytesCurBlock(toWrite);
packets.add(p);
{quote}
In {{DFSStripedOutputStream}}, have not calculated checksum for parity blocks
in {{DFSStripedOutputStream}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8231) StackTrace displayed at client while QuotaByStorageType exceeds
[ https://issues.apache.org/jira/browse/HDFS-8231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-8231: - Attachment: HDFS-8231.00.patch Thanks [~andreina] and [~cnauroth] for looking at the issue. I attached a patch for the fix. StackTrace displayed at client while QuotaByStorageType exceeds --- Key: HDFS-8231 URL: https://issues.apache.org/jira/browse/HDFS-8231 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Reporter: J.Andreina Assignee: J.Andreina Priority: Minor Attachments: HDFS-8231.00.patch StackTrace displayed at client while QuotaByStorageType exceeds. With reference to HDFS-2360, feel better to fix this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8213) DFSClient should use hdfs.client.htrace HTrace configuration prefix rather than hadoop.htrace
[ https://issues.apache.org/jira/browse/HDFS-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-8213: --- Assignee: Colin Patrick McCabe (was: Brahma Reddy Battula) Status: Patch Available (was: Open) DFSClient should use hdfs.client.htrace HTrace configuration prefix rather than hadoop.htrace - Key: HDFS-8213 URL: https://issues.apache.org/jira/browse/HDFS-8213 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.0 Reporter: Billie Rinaldi Assignee: Colin Patrick McCabe Priority: Critical Attachments: HDFS-8213.001.patch DFSClient initializing SpanReceivers is a problem for Accumulo, which manages SpanReceivers through its own configuration. This results in the same receivers being registered multiple times and spans being delivered more than once. The documentation says SpanReceiverHost.getInstance should be issued once per process, so there is no expectation that DFSClient should do this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8213) DFSClient should use hdfs.client.htrace HTrace configuration prefix rather than hadoop.htrace
[ https://issues.apache.org/jira/browse/HDFS-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-8213: --- Attachment: HDFS-8213.001.patch DFSClient should use hdfs.client.htrace HTrace configuration prefix rather than hadoop.htrace - Key: HDFS-8213 URL: https://issues.apache.org/jira/browse/HDFS-8213 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.0 Reporter: Billie Rinaldi Assignee: Brahma Reddy Battula Priority: Critical Attachments: HDFS-8213.001.patch DFSClient initializing SpanReceivers is a problem for Accumulo, which manages SpanReceivers through its own configuration. This results in the same receivers being registered multiple times and spans being delivered more than once. The documentation says SpanReceiverHost.getInstance should be issued once per process, so there is no expectation that DFSClient should do this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8233) Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when the last stripe is at the block group boundary
[
https://issues.apache.org/jira/browse/HDFS-8233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-8233:
Attachment: hdfs8233-HDFS-7285.000.patch
When we call {{getCurrentBlockGroupBytes}}, the block object in each streamer
cannot reflect the real size since we cannot guarantee that all the packets
have been sent out and acks have also been received. The {{bytesCurBlock}}
field can be set to 0 when an internal block is full. Thus it is hard to
compute the accurate block size at this time. However, for
{{writeParityCellsForLastStripe}} what we need is only the parity cell size
which can be computed based on {{bytesCurBlock}}.
The 000 patch fixes the issue based on the above theory. It also adds a new
unit test case which fails with original code.
Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when the last stripe is
at the block group boundary
Key: HDFS-8233
URL: https://issues.apache.org/jira/browse/HDFS-8233
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Jing Zhao
Assignee: Jing Zhao
Attachments: hdfs8233-HDFS-7285.000.patch
Currently {{DFSStripedOutputStream#getCurrentBlockGroupBytes}} simply uses
{{getBytesCurBlock}} of each streamer to calculate the block group size. This
is wrong when the last stripe is at the block group boundary, since the
{{bytesCurBlock}} is set to 0 if an internal block is finished.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-8223) Should calculate checksum for parity blocks in DFSStripedOutputStream
[
https://issues.apache.org/jira/browse/HDFS-8223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao resolved HDFS-8223.
-
Resolution: Fixed
Fix Version/s: HDFS-7285
Hadoop Flags: Reviewed
I've committed this to the feature branch.
Should calculate checksum for parity blocks in DFSStripedOutputStream
-
Key: HDFS-8223
URL: https://issues.apache.org/jira/browse/HDFS-8223
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Fix For: HDFS-7285
Attachments: HDFS-8223.001.patch, HDFS-8223.rebase.patch
{quote}
DFSPacket p = createPacket(packetSize, chunksPerPacket,
streamer.getBytesCurBlock(),
streamer.getAndIncCurrentSeqno(), false);
int maxBytesToPacket = p.getMaxChunks() * bytesPerChecksum;
int toWrite = byteBuffer.remaining() maxBytesToPacket ?
maxBytesToPacket: byteBuffer.remaining();
p.writeData(byteBuffer, toWrite);
streamer.incBytesCurBlock(toWrite);
packets.add(p);
{quote}
In {{DFSStripedOutputStream}}, have not calculated checksum for parity blocks
in {{DFSStripedOutputStream}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8215) Refactor NamenodeFsck#check method
[ https://issues.apache.org/jira/browse/HDFS-8215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-8215: -- Priority: Minor (was: Major) Hadoop Flags: Reviewed +1 the patch looks good. Thanks also for putting additional comments. Refactor NamenodeFsck#check method -- Key: HDFS-8215 URL: https://issues.apache.org/jira/browse/HDFS-8215 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Takanobu Asanuma Assignee: Takanobu Asanuma Priority: Minor Attachments: HDFS-8215.1.patch This method is very long, approximately 230 lines. Splitting this method into smaller methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8215) Refactor NamenodeFsck#check method
[ https://issues.apache.org/jira/browse/HDFS-8215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509845#comment-14509845 ] Hudson commented on HDFS-8215: -- FAILURE: Integrated in Hadoop-trunk-Commit #7654 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7654/]) HDFS-8215. Refactor NamenodeFsck#check method. Contributed by Takanobu Asanuma (szetszwo: rev ef4e9963b25d7d2e30f1071ddcaa9d92a7fe70f3) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Refactor NamenodeFsck#check method -- Key: HDFS-8215 URL: https://issues.apache.org/jira/browse/HDFS-8215 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Takanobu Asanuma Assignee: Takanobu Asanuma Priority: Minor Attachments: HDFS-8215.1.patch This method is very long, approximately 230 lines. Splitting this method into smaller methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8223) Should calculate checksum for parity blocks in DFSStripedOutputStream
[
https://issues.apache.org/jira/browse/HDFS-8223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-8223:
Attachment: HDFS-8223.rebase.patch
Upload the rebased patch.
Should calculate checksum for parity blocks in DFSStripedOutputStream
-
Key: HDFS-8223
URL: https://issues.apache.org/jira/browse/HDFS-8223
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-8223.001.patch, HDFS-8223.rebase.patch
{quote}
DFSPacket p = createPacket(packetSize, chunksPerPacket,
streamer.getBytesCurBlock(),
streamer.getAndIncCurrentSeqno(), false);
int maxBytesToPacket = p.getMaxChunks() * bytesPerChecksum;
int toWrite = byteBuffer.remaining() maxBytesToPacket ?
maxBytesToPacket: byteBuffer.remaining();
p.writeData(byteBuffer, toWrite);
streamer.incBytesCurBlock(toWrite);
packets.add(p);
{quote}
In {{DFSStripedOutputStream}}, have not calculated checksum for parity blocks
in {{DFSStripedOutputStream}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8215) Refactor NamenodeFsck#check method
[ https://issues.apache.org/jira/browse/HDFS-8215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-8215: -- Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) I have committed this. Thanks, Asanuma-san! Refactor NamenodeFsck#check method -- Key: HDFS-8215 URL: https://issues.apache.org/jira/browse/HDFS-8215 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Takanobu Asanuma Assignee: Takanobu Asanuma Priority: Minor Fix For: 2.8.0 Attachments: HDFS-8215.1.patch This method is very long, approximately 230 lines. Splitting this method into smaller methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-8233) Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when the last stripe is at the block group boundary
[
https://issues.apache.org/jira/browse/HDFS-8233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao resolved HDFS-8233.
-
Resolution: Fixed
Fix Version/s: HDFS-7285
Thanks for the review, Nicholas! I've committed this.
Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when the last stripe is
at the block group boundary
Key: HDFS-8233
URL: https://issues.apache.org/jira/browse/HDFS-8233
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Jing Zhao
Assignee: Jing Zhao
Fix For: HDFS-7285
Attachments: hdfs8233-HDFS-7285.000.patch
Currently {{DFSStripedOutputStream#getCurrentBlockGroupBytes}} simply uses
{{getBytesCurBlock}} of each streamer to calculate the block group size. This
is wrong when the last stripe is at the block group boundary, since the
{{bytesCurBlock}} is set to 0 if an internal block is finished.
For example, when the real block size is {{blockGroupSize - cellSize *
(numDataBlocks - 1)}}, i.e., the first internal block is full while the
others are not, the {{getCurrentBlockGroupBytes}} returns wrong result and
cause the write to fail.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6763) Initialize file system-wide quota once on transitioning to active
[
https://issues.apache.org/jira/browse/HDFS-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-6763:
-
Status: Patch Available (was: Open)
Initialize file system-wide quota once on transitioning to active
-
Key: HDFS-6763
URL: https://issues.apache.org/jira/browse/HDFS-6763
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, namenode
Reporter: Daryn Sharp
Assignee: Kihwal Lee
Attachments: HDFS-6763.patch
{{FSImage#loadEdits}} calls {{updateCountForQuota}} to recalculate verify
quotas for the entire namespace. A standby NN using shared edits calls this
method every minute. The standby may appear to hang for many seconds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8233) Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when the last stripe is at the block group boundary
[
https://issues.apache.org/jira/browse/HDFS-8233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-8233:
Description:
Currently {{DFSStripedOutputStream#getCurrentBlockGroupBytes}} simply uses
{{getBytesCurBlock}} of each streamer to calculate the block group size. This
is wrong when the last stripe is at the block group boundary, since the
{{bytesCurBlock}} is set to 0 if an internal block is finished.
For example, when the real block size is {{blockGroupSize - cellSize *
(numDataBlocks - 1)}}, i.e., the first internal block is full while the others
are not, the {{getCurrentBlockGroupBytes}} returns wrong result and cause the
write to fail.
was:Currently {{DFSStripedOutputStream#getCurrentBlockGroupBytes}} simply
uses {{getBytesCurBlock}} of each streamer to calculate the block group size.
This is wrong when the last stripe is at the block group boundary, since the
{{bytesCurBlock}} is set to 0 if an internal block is finished.
Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when the last stripe is
at the block group boundary
Key: HDFS-8233
URL: https://issues.apache.org/jira/browse/HDFS-8233
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Jing Zhao
Assignee: Jing Zhao
Attachments: hdfs8233-HDFS-7285.000.patch
Currently {{DFSStripedOutputStream#getCurrentBlockGroupBytes}} simply uses
{{getBytesCurBlock}} of each streamer to calculate the block group size. This
is wrong when the last stripe is at the block group boundary, since the
{{bytesCurBlock}} is set to 0 if an internal block is finished.
For example, when the real block size is {{blockGroupSize - cellSize *
(numDataBlocks - 1)}}, i.e., the first internal block is full while the
others are not, the {{getCurrentBlockGroupBytes}} returns wrong result and
cause the write to fail.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8236) Merge HDFS-8227 into EC branch
Haohui Mai created HDFS-8236: Summary: Merge HDFS-8227 into EC branch Key: HDFS-8236 URL: https://issues.apache.org/jira/browse/HDFS-8236 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Haohui Mai Assignee: Haohui Mai Attachments: HDFS-8236.000.patch This jira proposes to merge the changes proposed in HDFS-8227 into the EC branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8236) Merge HDFS-8227 into EC branch
[ https://issues.apache.org/jira/browse/HDFS-8236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-8236: - Attachment: HDFS-8236.000.patch Merge HDFS-8227 into EC branch -- Key: HDFS-8236 URL: https://issues.apache.org/jira/browse/HDFS-8236 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Haohui Mai Assignee: Haohui Mai Attachments: HDFS-8236.000.patch This jira proposes to merge the changes proposed in HDFS-8227 into the EC branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8233) Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when the last stripe is at the block group boundary
[
https://issues.apache.org/jira/browse/HDFS-8233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-8233:
--
Hadoop Flags: Reviewed
+1 patch looks good.
Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when the last stripe is
at the block group boundary
Key: HDFS-8233
URL: https://issues.apache.org/jira/browse/HDFS-8233
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Jing Zhao
Assignee: Jing Zhao
Attachments: hdfs8233-HDFS-7285.000.patch
Currently {{DFSStripedOutputStream#getCurrentBlockGroupBytes}} simply uses
{{getBytesCurBlock}} of each streamer to calculate the block group size. This
is wrong when the last stripe is at the block group boundary, since the
{{bytesCurBlock}} is set to 0 if an internal block is finished.
For example, when the real block size is {{blockGroupSize - cellSize *
(numDataBlocks - 1)}}, i.e., the first internal block is full while the
others are not, the {{getCurrentBlockGroupBytes}} returns wrong result and
cause the write to fail.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8228) TestDFSStripedOutputStream#testFileSmallerThanOneStripe2 failed
[
https://issues.apache.org/jira/browse/HDFS-8228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509979#comment-14509979
]
Tsz Wo Nicholas Sze commented on HDFS-8228:
---
Just tried. It did not fail in my machine.
TestDFSStripedOutputStream#testFileSmallerThanOneStripe2 failed
---
Key: HDFS-8228
URL: https://issues.apache.org/jira/browse/HDFS-8228
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Li Bo
Fix For: HDFS-7285
Playing with the branch, found below:
{noformat}
java.io.EOFException: Premature EOF: no length prefix available
at
org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2343)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:244)
at
org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:823)
2015-04-23 23:21:08,666 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped
SelectChannelConnector@localhost:0
2015-04-23 23:21:08,769 INFO ipc.Server (Server.java:stop(2540)) - Stopping
server on 57920
2015-04-23 23:21:08,770 INFO datanode.DataNode
(BlockReceiver.java:receiveBlock(826)) - Exception for
BP-1850767374-10.239.12.51-1429802363548:blk_-9223372036854775737_1007
java.io.InterruptedIOException: Interrupted while waiting for IO on channel
java.nio.channels.SocketChannel[closed]. 6 millis timeout left.
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:201)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:472)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:787)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:793)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
at java.lang.Thread.run(Thread.java:745)
2015-04-23 23:21:08,769 INFO datanode.DataNode
(BlockReceiver.java:run(1250)) - PacketResponder:
BP-1850767374-10.239.12.51-1429802363548:blk_-9223372036854775737_1007,
type=LAST_IN_PIPELINE, downstreams=0:[]: Thread is interrupted.
2015-04-23 23:21:08,776 WARN datanode.DataNode
(BPServiceActor.java:offerService(756)) - BPOfferService for Block pool
BP-1850767374-10.239.12.51-1429802363548 (Datanode Uuid
72b12e39-77cb-463d-a919-0ac06d166fcd) service to localhost/127.0.0.1:36877
interrupted
2015-04-23 23:21:08,776 INFO ipc.Server (Server.java:run(846)) - Stopping
IPC Server Responder
{noformat}
[~libo-intel]], would you help with this?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8211) DataNode UUID is always null in the JMX counter
[
https://issues.apache.org/jira/browse/HDFS-8211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509367#comment-14509367
]
Arpit Agarwal commented on HDFS-8211:
-
bq. The reason being that all the functions of DataNode are private
Hi Anu, we often work around this by making the method package private and
tagging it as {{@VisibleForTesting}}.
DataNode UUID is always null in the JMX counter
---
Key: HDFS-8211
URL: https://issues.apache.org/jira/browse/HDFS-8211
Project: Hadoop HDFS
Issue Type: Bug
Components: HDFS
Affects Versions: 2.7.0
Reporter: Anu Engineer
Assignee: Anu Engineer
Priority: Minor
Attachments: hdfs-8211.001.patch
The DataNode JMX counters are tagged with DataNode UUID, but it always gets a
null value instead of the UUID.
{code}
Hadoop:service=DataNode,name=FSDatasetState*-null*.
{code}
This null is supposed be the datanode UUID.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8147) Mover should not schedule two replicas to the same DN storage
[
https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509401#comment-14509401
]
Hadoop QA commented on HDFS-8147:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 37s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | javac | 7m 26s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 34s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 7m 47s | The applied patch generated 1
additional checkstyle issues. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 4s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | native | 3m 12s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 186m 38s | Tests passed in hadoop-hdfs.
|
| | | 234m 50s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12727606/HDFS-8147_4.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / baf8bc6 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/10358/artifact/patchprocess/checkstyle-result-diff.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/10358/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/10358/testReport/ |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10358/console |
This message was automatically generated.
Mover should not schedule two replicas to the same DN storage
-
Key: HDFS-8147
URL: https://issues.apache.org/jira/browse/HDFS-8147
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer mover
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch,
HDFS-8147_3.patch, HDFS-8147_4.patch
*Scenario:*
1. Three DN cluster. For DNs storage type is like this.
DN1 : DISK,ARCHIVE
DN2 : DISK
DN3 : DISK,ARCHIVE (All DNs are in same rack)
2. One file with two replicas (In DN1 and DN2)
3. Set file storage policy COLD
4. Now execute Mover.
*Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE
*Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to
D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is
selecting again DN1:ARCHIVE for target where already same block exists.
{{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the
node as target where already same replica exists. The dispatcher will fail
to move block as shown in the log below. Then, the Mover will try again in
next iteration.
*Logs*
{code}
15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move
blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to
10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status
message opReplaceBlock
BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received
exception
org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica
FinalizedReplica, blk_1073741852_1028, FINALIZED
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6763) Initialize file system-wide quota once on transitioning to active
[
https://issues.apache.org/jira/browse/HDFS-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-6763:
-
Summary: Initialize file system-wide quota once on transitioning to active
(was: Perform file system-wide quota initialization once on transitioning to
active)
Initialize file system-wide quota once on transitioning to active
-
Key: HDFS-6763
URL: https://issues.apache.org/jira/browse/HDFS-6763
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, namenode
Reporter: Daryn Sharp
Assignee: Kihwal Lee
Attachments: HDFS-6763.patch
{{FSImage#loadEdits}} calls {{updateCountForQuota}} to recalculate verify
quotas for the entire namespace. A standby NN using shared edits calls this
method every minute. The standby may appear to hang for many seconds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8213) DFSClient should not instantiate SpanReceiverHost
[
https://issues.apache.org/jira/browse/HDFS-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509472#comment-14509472
]
Colin Patrick McCabe commented on HDFS-8213:
bq. can you people suggest configuration for DFSClient..?
I'm thinking {{hdfs.client.htrace.spanreceiver.classes}}. It's not completely
trivial because I have to change our SpanReceiverHost thing, but shouldn't be
too bad... let me see if I can post the patch
DFSClient should not instantiate SpanReceiverHost
-
Key: HDFS-8213
URL: https://issues.apache.org/jira/browse/HDFS-8213
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Billie Rinaldi
Assignee: Brahma Reddy Battula
Priority: Critical
DFSClient initializing SpanReceivers is a problem for Accumulo, which manages
SpanReceivers through its own configuration. This results in the same
receivers being registered multiple times and spans being delivered more than
once. The documentation says SpanReceiverHost.getInstance should be issued
once per process, so there is no expectation that DFSClient should do this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7281) Missing block is marked as corrupted block
[
https://issues.apache.org/jira/browse/HDFS-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14510451#comment-14510451
]
Yongjun Zhang commented on HDFS-7281:
-
Hi [~mingma],
Thanks for the new rev and sorry for being late. It looks good, only one
comment here:
We may change:
{code}
if (missing 0) {
out.print( MISSING + missing + blocks of total size + missize +
B\n);
}
if (corrupt 0) {
out.print( CORRUPT + corrupt + blocks of total size +
corruptSize + B\n);
}
if (underReplicatedPerFile == 0 misReplicatedPerFile == 0) {
out.print( OK\n);
}
{code}
to
{code}
if (missing 0 || corrupt 0) {
if (missing 0) {
out.print( MISSING + missing + blocks of total size + missize
+ B\n);
}
if (corrupt 0) {
out.print( CORRUPT + corrupt + blocks of total size +
corruptSize + B\n);
}
} else if (underReplicatedPerFile == 0 misReplicatedPerFile == 0) {
out.print( OK\n);
}
{code}
such that if there are missing/corrupt blocks, we don't report OK. Right?
BTW, Would you please write a release notes about this fsck report change?
Thanks.
Missing block is marked as corrupted block
--
Key: HDFS-7281
URL: https://issues.apache.org/jira/browse/HDFS-7281
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Ming Ma
Assignee: Ming Ma
Labels: supportability
Attachments: HDFS-7281-2.patch, HDFS-7281-3.patch, HDFS-7281-4.patch,
HDFS-7281.patch
In the situation where the block lost all its replicas, fsck shows the block
is missing as well as corrupted. Perhaps it is better not to mark the block
corrupted in this case. The reason it is marked as corrupted is
numCorruptNodes == numNodes == 0 in the following code.
{noformat}
BlockManager
final boolean isCorrupt = numCorruptNodes == numNodes;
{noformat}
Would like to clarify if it is the intent to mark missing block as corrupted
or it is just a bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8147) Mover should not schedule two replicas to the same DN storage
[
https://issues.apache.org/jira/browse/HDFS-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-8147:
--
Resolution: Fixed
Fix Version/s: 2.7.1
Status: Resolved (was: Patch Available)
I have committed this. Thanks, surendra!
Mover should not schedule two replicas to the same DN storage
-
Key: HDFS-8147
URL: https://issues.apache.org/jira/browse/HDFS-8147
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer mover
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
Fix For: 2.7.1
Attachments: HDFS-8147.patch, HDFS-8147_1.patch, HDFS-8147_2.patch,
HDFS-8147_3.patch, HDFS-8147_4.patch
*Scenario:*
1. Three DN cluster. For DNs storage type is like this.
DN1 : DISK,ARCHIVE
DN2 : DISK
DN3 : DISK,ARCHIVE (All DNs are in same rack)
2. One file with two replicas (In DN1 and DN2)
3. Set file storage policy COLD
4. Now execute Mover.
*Expected Result:* File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE
*Actual Result:* {{chooseTargetInSameNode()}} move D1:DISK block to
D1:ARCHIVE, but in next iteration {{chooseTarget()}} for same rake is
selecting again DN1:ARCHIVE for target where already same block exists.
{{chooseTargetInSameNode()}} and {{chooseTarget()}} should not select the
node as target where already same replica exists. The dispatcher will fail
to move block as shown in the log below. Then, the Mover will try again in
next iteration.
*Logs*
{code}
15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move
blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to
10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status
message opReplaceBlock
BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received
exception
org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica
FinalizedReplica, blk_1073741852_1028, FINALIZED
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8228) TestDFSStripedOutputStream#testFileSmallerThanOneStripe2 failed
[
https://issues.apache.org/jira/browse/HDFS-8228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-8228:
Summary: TestDFSStripedOutputStream#testFileSmallerThanOneStripe2 failed
(was: TestDFSStripedInputStream#testFileSmallerThanOneStripe2 failed)
TestDFSStripedOutputStream#testFileSmallerThanOneStripe2 failed
---
Key: HDFS-8228
URL: https://issues.apache.org/jira/browse/HDFS-8228
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Li Bo
Fix For: HDFS-7285
Playing with the branch, found below:
{noformat}
java.io.EOFException: Premature EOF: no length prefix available
at
org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2343)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:244)
at
org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:823)
2015-04-23 23:21:08,666 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped
SelectChannelConnector@localhost:0
2015-04-23 23:21:08,769 INFO ipc.Server (Server.java:stop(2540)) - Stopping
server on 57920
2015-04-23 23:21:08,770 INFO datanode.DataNode
(BlockReceiver.java:receiveBlock(826)) - Exception for
BP-1850767374-10.239.12.51-1429802363548:blk_-9223372036854775737_1007
java.io.InterruptedIOException: Interrupted while waiting for IO on channel
java.nio.channels.SocketChannel[closed]. 6 millis timeout left.
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:201)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:472)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:787)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:793)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
at java.lang.Thread.run(Thread.java:745)
2015-04-23 23:21:08,769 INFO datanode.DataNode
(BlockReceiver.java:run(1250)) - PacketResponder:
BP-1850767374-10.239.12.51-1429802363548:blk_-9223372036854775737_1007,
type=LAST_IN_PIPELINE, downstreams=0:[]: Thread is interrupted.
2015-04-23 23:21:08,776 WARN datanode.DataNode
(BPServiceActor.java:offerService(756)) - BPOfferService for Block pool
BP-1850767374-10.239.12.51-1429802363548 (Datanode Uuid
72b12e39-77cb-463d-a919-0ac06d166fcd) service to localhost/127.0.0.1:36877
interrupted
2015-04-23 23:21:08,776 INFO ipc.Server (Server.java:run(846)) - Stopping
IPC Server Responder
{noformat}
[~libo-intel]], would you help with this?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8228) TestDFSStripedInputStream#testFileSmallerThanOneStripe2 failed
[
https://issues.apache.org/jira/browse/HDFS-8228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-8228:
Issue Type: Sub-task (was: Bug)
Parent: HDFS-7285
TestDFSStripedInputStream#testFileSmallerThanOneStripe2 failed
--
Key: HDFS-8228
URL: https://issues.apache.org/jira/browse/HDFS-8228
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Li Bo
Fix For: HDFS-7285
Playing with the branch, found below:
{noformat}
java.io.EOFException: Premature EOF: no length prefix available
at
org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2343)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:244)
at
org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:823)
2015-04-23 23:21:08,666 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped
SelectChannelConnector@localhost:0
2015-04-23 23:21:08,769 INFO ipc.Server (Server.java:stop(2540)) - Stopping
server on 57920
2015-04-23 23:21:08,770 INFO datanode.DataNode
(BlockReceiver.java:receiveBlock(826)) - Exception for
BP-1850767374-10.239.12.51-1429802363548:blk_-9223372036854775737_1007
java.io.InterruptedIOException: Interrupted while waiting for IO on channel
java.nio.channels.SocketChannel[closed]. 6 millis timeout left.
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:201)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:472)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:787)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:793)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
at java.lang.Thread.run(Thread.java:745)
2015-04-23 23:21:08,769 INFO datanode.DataNode
(BlockReceiver.java:run(1250)) - PacketResponder:
BP-1850767374-10.239.12.51-1429802363548:blk_-9223372036854775737_1007,
type=LAST_IN_PIPELINE, downstreams=0:[]: Thread is interrupted.
2015-04-23 23:21:08,776 WARN datanode.DataNode
(BPServiceActor.java:offerService(756)) - BPOfferService for Block pool
BP-1850767374-10.239.12.51-1429802363548 (Datanode Uuid
72b12e39-77cb-463d-a919-0ac06d166fcd) service to localhost/127.0.0.1:36877
interrupted
2015-04-23 23:21:08,776 INFO ipc.Server (Server.java:run(846)) - Stopping
IPC Server Responder
{noformat}
[~libo-intel]], would you help with this?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8234) DistributedFileSystem should override listStatus(Path f, PathFilter filter)
Rohini Palaniswamy created HDFS-8234: Summary: DistributedFileSystem should override listStatus(Path f, PathFilter filter) Key: HDFS-8234 URL: https://issues.apache.org/jira/browse/HDFS-8234 Project: Hadoop HDFS Issue Type: Improvement Reporter: Rohini Palaniswamy HDFS-985 added partial listing in listStatus to avoid listing entries of large directory in one go. If listStatus(Path p, PathFilter f) call is made, filter is applied after fetching all the entries resulting in a big list being constructed on the client side. If the DistributedFileSystem.listStatusInternal() applied the PathFilter it would be more efficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8234) DistributedFileSystem should override listStatus(Path f, PathFilter filter)
[ https://issues.apache.org/jira/browse/HDFS-8234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohini Palaniswamy updated HDFS-8234: - Labels: newbie (was: ) DistributedFileSystem should override listStatus(Path f, PathFilter filter) Key: HDFS-8234 URL: https://issues.apache.org/jira/browse/HDFS-8234 Project: Hadoop HDFS Issue Type: Improvement Reporter: Rohini Palaniswamy Labels: newbie HDFS-985 added partial listing in listStatus to avoid listing entries of large directory in one go. If listStatus(Path p, PathFilter f) call is made, filter is applied after fetching all the entries resulting in a big list being constructed on the client side. If the DistributedFileSystem.listStatusInternal() applied the PathFilter it would be more efficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8236) Merge HDFS-8227 into EC branch
[ https://issues.apache.org/jira/browse/HDFS-8236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14510030#comment-14510030 ] Tsz Wo Nicholas Sze commented on HDFS-8236: --- In the EC branch, BlockInfoContiguous is divided to two BlockInfo and BlockInfoContiguous. How about we also do the same in HDFS-8227? Merge HDFS-8227 into EC branch -- Key: HDFS-8236 URL: https://issues.apache.org/jira/browse/HDFS-8236 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Haohui Mai Assignee: Haohui Mai Attachments: HDFS-8236.000.patch This jira proposes to merge the changes proposed in HDFS-8227 into the EC branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8228) TestDFSStripedInputStream#testFileSmallerThanOneStripe2 failed
[
https://issues.apache.org/jira/browse/HDFS-8228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-8228:
Summary: TestDFSStripedInputStream#testFileSmallerThanOneStripe2 failed
(was: TestDFSStripedOutputStream#testFileSmallerThanOneStripe2 failed)
TestDFSStripedInputStream#testFileSmallerThanOneStripe2 failed
--
Key: HDFS-8228
URL: https://issues.apache.org/jira/browse/HDFS-8228
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Li Bo
Fix For: HDFS-7285
Playing with the branch, found below:
{noformat}
java.io.EOFException: Premature EOF: no length prefix available
at
org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2343)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:244)
at
org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:823)
2015-04-23 23:21:08,666 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped
SelectChannelConnector@localhost:0
2015-04-23 23:21:08,769 INFO ipc.Server (Server.java:stop(2540)) - Stopping
server on 57920
2015-04-23 23:21:08,770 INFO datanode.DataNode
(BlockReceiver.java:receiveBlock(826)) - Exception for
BP-1850767374-10.239.12.51-1429802363548:blk_-9223372036854775737_1007
java.io.InterruptedIOException: Interrupted while waiting for IO on channel
java.nio.channels.SocketChannel[closed]. 6 millis timeout left.
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:201)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:472)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:787)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:793)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
at java.lang.Thread.run(Thread.java:745)
2015-04-23 23:21:08,769 INFO datanode.DataNode
(BlockReceiver.java:run(1250)) - PacketResponder:
BP-1850767374-10.239.12.51-1429802363548:blk_-9223372036854775737_1007,
type=LAST_IN_PIPELINE, downstreams=0:[]: Thread is interrupted.
2015-04-23 23:21:08,776 WARN datanode.DataNode
(BPServiceActor.java:offerService(756)) - BPOfferService for Block pool
BP-1850767374-10.239.12.51-1429802363548 (Datanode Uuid
72b12e39-77cb-463d-a919-0ac06d166fcd) service to localhost/127.0.0.1:36877
interrupted
2015-04-23 23:21:08,776 INFO ipc.Server (Server.java:run(846)) - Stopping
IPC Server Responder
{noformat}
[~libo-intel]], would you help with this?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8191) Fix byte to integer casting in SimulatedFSDataset#simulatedByte
[ https://issues.apache.org/jira/browse/HDFS-8191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14510048#comment-14510048 ] Andrew Wang commented on HDFS-8191: --- I kicked the precommit, dunno why it didn't run. https://builds.apache.org/job/PreCommit-HDFS-Build/10363/ Fix byte to integer casting in SimulatedFSDataset#simulatedByte --- Key: HDFS-8191 URL: https://issues.apache.org/jira/browse/HDFS-8191 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.0 Reporter: Zhe Zhang Assignee: Zhe Zhang Priority: Minor Attachments: HDFS-8191.000.patch, HDFS-8191.001.patch, HDFS-8191.002.patch, HDFS-8191.003.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8209) Support different number of datanode directories in MiniDFSCluster.
[
https://issues.apache.org/jira/browse/HDFS-8209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-8209:
--
Priority: Minor (was: Major)
Issue Type: Improvement (was: Bug)
Summary: Support different number of datanode directories in
MiniDFSCluster. (was: ArrayIndexOutOfBoundsException in MiniDFSCluster.)
Sure, we could support it.
(Revised Summary.)
Support different number of datanode directories in MiniDFSCluster.
---
Key: HDFS-8209
URL: https://issues.apache.org/jira/browse/HDFS-8209
Project: Hadoop HDFS
Issue Type: Improvement
Components: test
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
Priority: Minor
I want to create MiniDFSCluster with 2 datanode and for each datanode I want
to set different number of StorageTypes, but in this case I am getting
ArrayIndexOutOfBoundsException.
My cluster schema is like this.
{code}
final MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
.numDataNodes(2)
.storageTypes(new StorageType[][] {{
StorageType.DISK, StorageType.ARCHIVE },{ StorageType.DISK } })
.build();
{code}
*Exception* :
{code}
java.lang.ArrayIndexOutOfBoundsException: 1
at
org.apache.hadoop.hdfs.MiniDFSCluster.makeDataNodeDirs(MiniDFSCluster.java:1218)
at
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1402)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:832)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8052) Move WebHdfsFileSystem into hadoop-hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-8052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14510159#comment-14510159 ] Hudson commented on HDFS-8052: -- SUCCESS: Integrated in Hadoop-trunk-Commit #7657 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7657/]) HDFS-8052. Move WebHdfsFileSystem into hadoop-hdfs-client. Contributed by Haohui Mai. (wheat9: rev bcf89ddc7d52e04725caf104f5958e33d9f51b35) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryUtils.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/ByteRangeInputStream.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/TokenAspect.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/TokenAspect.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/ByteRangeInputStream.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstantsClient.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/resources/BufferSizeParam.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockStorageLocationUtil.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/KerberosUgiAuthenticator.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/URLConnectionFactory.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsLocatedFileStatus.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/KerberosUgiAuthenticator.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/SWebHdfsFileSystem.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/SWebHdfsFileSystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/BufferSizeParam.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/URLConnectionFactory.java Move WebHdfsFileSystem into hadoop-hdfs-client -- Key: HDFS-8052 URL: https://issues.apache.org/jira/browse/HDFS-8052 Project: Hadoop HDFS Issue Type: Sub-task Components: build Reporter: Haohui Mai Assignee: Haohui Mai Fix For: 2.8.0 Attachments: HDFS-8052.000.patch, HDFS-8052.001.patch, HDFS-8052.002.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8052) Move WebHdfsFileSystem into hadoop-hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-8052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-8052: - Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) I've committed the patch to trunk and branch-2. Thanks Nicholas for the reviews. Move WebHdfsFileSystem into hadoop-hdfs-client -- Key: HDFS-8052 URL: https://issues.apache.org/jira/browse/HDFS-8052 Project: Hadoop HDFS Issue Type: Sub-task Components: build Reporter: Haohui Mai Assignee: Haohui Mai Fix For: 2.8.0 Attachments: HDFS-8052.000.patch, HDFS-8052.001.patch, HDFS-8052.002.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8237) Move all protocol classes used by ClientProtocol to hdfs-client
Haohui Mai created HDFS-8237: Summary: Move all protocol classes used by ClientProtocol to hdfs-client Key: HDFS-8237 URL: https://issues.apache.org/jira/browse/HDFS-8237 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Haohui Mai Assignee: Haohui Mai Attachments: HDFS-8237.000.patch This jira proposes to move the classes in the hdfs project referred by ClientProtocol into the hdfs-client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8237) Move all protocol classes used by ClientProtocol to hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-8237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-8237: - Attachment: HDFS-8237.000.patch Move all protocol classes used by ClientProtocol to hdfs-client --- Key: HDFS-8237 URL: https://issues.apache.org/jira/browse/HDFS-8237 Project: Hadoop HDFS Issue Type: Sub-task Components: build Reporter: Haohui Mai Assignee: Haohui Mai Attachments: HDFS-8237.000.patch This jira proposes to move the classes in the hdfs project referred by ClientProtocol into the hdfs-client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8237) Move all protocol classes used by ClientProtocol to hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-8237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-8237: - Status: Patch Available (was: Open) Move all protocol classes used by ClientProtocol to hdfs-client --- Key: HDFS-8237 URL: https://issues.apache.org/jira/browse/HDFS-8237 Project: Hadoop HDFS Issue Type: Sub-task Components: build Reporter: Haohui Mai Assignee: Haohui Mai Attachments: HDFS-8237.000.patch This jira proposes to move the classes in the hdfs project referred by ClientProtocol into the hdfs-client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8228) TestDFSStripedInputStream#testFileSmallerThanOneStripe2 failed
[
https://issues.apache.org/jira/browse/HDFS-8228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14510184#comment-14510184
]
Kai Zheng commented on HDFS-8228:
-
I reassigned to Jing. Thanks Jing for the taking. [~libo-intel] please comment
if you have some finding, thanks.
TestDFSStripedInputStream#testFileSmallerThanOneStripe2 failed
--
Key: HDFS-8228
URL: https://issues.apache.org/jira/browse/HDFS-8228
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Jing Zhao
Fix For: HDFS-7285
Playing with the branch, found below:
{noformat}
java.io.EOFException: Premature EOF: no length prefix available
at
org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2343)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:244)
at
org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:823)
2015-04-23 23:21:08,666 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped
SelectChannelConnector@localhost:0
2015-04-23 23:21:08,769 INFO ipc.Server (Server.java:stop(2540)) - Stopping
server on 57920
2015-04-23 23:21:08,770 INFO datanode.DataNode
(BlockReceiver.java:receiveBlock(826)) - Exception for
BP-1850767374-10.239.12.51-1429802363548:blk_-9223372036854775737_1007
java.io.InterruptedIOException: Interrupted while waiting for IO on channel
java.nio.channels.SocketChannel[closed]. 6 millis timeout left.
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:201)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:472)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:787)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:793)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
at java.lang.Thread.run(Thread.java:745)
2015-04-23 23:21:08,769 INFO datanode.DataNode
(BlockReceiver.java:run(1250)) - PacketResponder:
BP-1850767374-10.239.12.51-1429802363548:blk_-9223372036854775737_1007,
type=LAST_IN_PIPELINE, downstreams=0:[]: Thread is interrupted.
2015-04-23 23:21:08,776 WARN datanode.DataNode
(BPServiceActor.java:offerService(756)) - BPOfferService for Block pool
BP-1850767374-10.239.12.51-1429802363548 (Datanode Uuid
72b12e39-77cb-463d-a919-0ac06d166fcd) service to localhost/127.0.0.1:36877
interrupted
2015-04-23 23:21:08,776 INFO ipc.Server (Server.java:run(846)) - Stopping
IPC Server Responder
{noformat}
[~libo-intel]], would you help with this?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14510191#comment-14510191 ] Kai Zheng commented on HDFS-8033: - Good work here. Looks like the Jenkins building works now? Erasure coding: stateful (non-positional) read from files in striped layout --- Key: HDFS-8033 URL: https://issues.apache.org/jira/browse/HDFS-8033 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: HDFS-7285 Reporter: Zhe Zhang Assignee: Zhe Zhang Attachments: ByteBufferStrategy_support_len.patch, HDFS-8033.000.patch, HDFS-8033.001.patch, HDFS-8033.002.patch, HDFS-8033.003.patch, hdfs8033-HDFS-7285.04.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7995) Implement chmod in the HDFS Web UI
[ https://issues.apache.org/jira/browse/HDFS-7995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14510206#comment-14510206 ] Haohui Mai commented on HDFS-7995: -- Sorry for the delay. It looks to me that it is somewhat noisy from the UX prospective as it needs a big modal dialog to update the permission. What about letting the user to update the permission string in place through X-editable? Implement chmod in the HDFS Web UI -- Key: HDFS-7995 URL: https://issues.apache.org/jira/browse/HDFS-7995 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Ravi Prakash Assignee: Ravi Prakash Attachments: HDFS-7995.01.patch We should let users change the permissions of files and directories using the HDFS Web UI -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8217) During block recovery for truncate Log new Block Id in case of copy-on-truncate is true.
[
https://issues.apache.org/jira/browse/HDFS-8217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14510219#comment-14510219
]
Konstantin Shvachko commented on HDFS-8217:
---
I think it is fine to ignore AvoidInlineConditionalsCheck.
Automation tools should help improve code, but this one just forces you to
write bunch of code for a simple LOG message.
+1 on your first version.
During block recovery for truncate Log new Block Id in case of
copy-on-truncate is true.
Key: HDFS-8217
URL: https://issues.apache.org/jira/browse/HDFS-8217
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Reporter: Vinayakumar B
Assignee: Vinayakumar B
Attachments: HDFS-8217-01.patch, HDFS-8217-02.patch
During the block recovery for the truncate, include the newblock Id in case
copy-on-truncate set to true.
Current log message is just as below.
{noformat}NameNode at /127.0.0.1:55036 calls
recoverBlock(BP-409377067-xx.xx.xx.253-1429679063755:blk_1073741825_1001,
targets=[DatanodeInfoWithStorage[127.0.0.1:55045,null,null]],
newGenerationStamp=1002){noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8213) DFSClient should use hdfs.client.htrace HTrace configuration prefix rather than hadoop.htrace
[
https://issues.apache.org/jira/browse/HDFS-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14510323#comment-14510323
]
Hadoop QA commented on HDFS-8213:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 30s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | javac | 7m 25s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 7m 41s | The applied patch generated 1
additional checkstyle issues. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 44s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | common tests | 22m 50s | Tests passed in
hadoop-common. |
| {color:green}+1{color} | hdfs tests | 167m 16s | Tests passed in hadoop-hdfs.
|
| | | 236m 34s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12727720/HDFS-8213.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / ef4e996 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/10361/artifact/patchprocess/checkstyle-result-diff.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/10361/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/10361/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/10361/testReport/ |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10361/console |
This message was automatically generated.
DFSClient should use hdfs.client.htrace HTrace configuration prefix rather
than hadoop.htrace
-
Key: HDFS-8213
URL: https://issues.apache.org/jira/browse/HDFS-8213
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Billie Rinaldi
Assignee: Colin Patrick McCabe
Priority: Critical
Attachments: HDFS-8213.001.patch
DFSClient initializing SpanReceivers is a problem for Accumulo, which manages
SpanReceivers through its own configuration. This results in the same
receivers being registered multiple times and spans being delivered more than
once. The documentation says SpanReceiverHost.getInstance should be issued
once per process, so there is no expectation that DFSClient should do this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8228) TestDFSStripedInputStream#testFileSmallerThanOneStripe2 failed
[
https://issues.apache.org/jira/browse/HDFS-8228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14510063#comment-14510063
]
Jing Zhao commented on HDFS-8228:
-
bq. It did not fail in my machine.
My bad. I thought the failure was in TestDFSStripedOutputStream, but it is
actually in TestDFSStripedInputStream. And I can reproduce it in my local run.
TestDFSStripedInputStream#testFileSmallerThanOneStripe2 failed
--
Key: HDFS-8228
URL: https://issues.apache.org/jira/browse/HDFS-8228
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Li Bo
Fix For: HDFS-7285
Playing with the branch, found below:
{noformat}
java.io.EOFException: Premature EOF: no length prefix available
at
org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2343)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:244)
at
org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:823)
2015-04-23 23:21:08,666 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped
SelectChannelConnector@localhost:0
2015-04-23 23:21:08,769 INFO ipc.Server (Server.java:stop(2540)) - Stopping
server on 57920
2015-04-23 23:21:08,770 INFO datanode.DataNode
(BlockReceiver.java:receiveBlock(826)) - Exception for
BP-1850767374-10.239.12.51-1429802363548:blk_-9223372036854775737_1007
java.io.InterruptedIOException: Interrupted while waiting for IO on channel
java.nio.channels.SocketChannel[closed]. 6 millis timeout left.
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:201)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:472)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:787)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:793)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
at java.lang.Thread.run(Thread.java:745)
2015-04-23 23:21:08,769 INFO datanode.DataNode
(BlockReceiver.java:run(1250)) - PacketResponder:
BP-1850767374-10.239.12.51-1429802363548:blk_-9223372036854775737_1007,
type=LAST_IN_PIPELINE, downstreams=0:[]: Thread is interrupted.
2015-04-23 23:21:08,776 WARN datanode.DataNode
(BPServiceActor.java:offerService(756)) - BPOfferService for Block pool
BP-1850767374-10.239.12.51-1429802363548 (Datanode Uuid
72b12e39-77cb-463d-a919-0ac06d166fcd) service to localhost/127.0.0.1:36877
interrupted
2015-04-23 23:21:08,776 INFO ipc.Server (Server.java:run(846)) - Stopping
IPC Server Responder
{noformat}
[~libo-intel]], would you help with this?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8052) Move WebHdfsFileSystem into hadoop-hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-8052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-8052: -- Hadoop Flags: Reviewed +1 patch looks good. Move WebHdfsFileSystem into hadoop-hdfs-client -- Key: HDFS-8052 URL: https://issues.apache.org/jira/browse/HDFS-8052 Project: Hadoop HDFS Issue Type: Sub-task Components: build Reporter: Haohui Mai Assignee: Haohui Mai Attachments: HDFS-8052.000.patch, HDFS-8052.001.patch, HDFS-8052.002.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8033: Status: Open (was: Patch Available) Erasure coding: stateful (non-positional) read from files in striped layout --- Key: HDFS-8033 URL: https://issues.apache.org/jira/browse/HDFS-8033 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: HDFS-7285 Reporter: Zhe Zhang Assignee: Zhe Zhang Attachments: ByteBufferStrategy_support_len.patch, HDFS-8033.000.patch, HDFS-8033.001.patch, HDFS-8033.002.patch, HDFS-8033.003.patch, hdfs8033-HDFS-7285.04.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8238) Move ClientProtocol to the hdfs-client
Haohui Mai created HDFS-8238:
Summary: Move ClientProtocol to the hdfs-client
Key: HDFS-8238
URL: https://issues.apache.org/jira/browse/HDFS-8238
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Takanobu Asanuma
The {{ClientProtocol}} class defines the RPC protocol between the NN and the
client. This jira proposes to move it into the hdfs-client module.
The jira needs to move:
* {{o.a.h.hdfs.server.namenode.SafeModeException}} and
{{o.a.h.hdfs.server.namenode.NotReplicatedYetException}} to the hdfs-client
package
* Remove the reference of {{DistributedFileSystem}} in the javadoc
* Create a copy of {{DFSConfigKeys.DFS_NAMENODE_KERBEROS_PRINCIPAL_KEY}} in
{{HdfsClientConfigKeys}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8019) Erasure Coding: erasure coding chunk buffer allocation and management
[ https://issues.apache.org/jira/browse/HDFS-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-8019: Description: As a task of HDFS-7344, this is to come up a chunk buffer pool allocating and managing coding chunk buffers, either based on on-heap or off-heap. Note this assumes some DataNodes are powerful in computing and performing EC coding work, so better to have this dedicated buffer pool and management. (was: As a task of HDFS-7344, this is to come up a chunk buffer pool allocating and managing coding chunk buffers, either based on on-heap or off-heap.) Erasure Coding: erasure coding chunk buffer allocation and management - Key: HDFS-8019 URL: https://issues.apache.org/jira/browse/HDFS-8019 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Vinayakumar B As a task of HDFS-7344, this is to come up a chunk buffer pool allocating and managing coding chunk buffers, either based on on-heap or off-heap. Note this assumes some DataNodes are powerful in computing and performing EC coding work, so better to have this dedicated buffer pool and management. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6763) Initialize file system-wide quota once on transitioning to active
[
https://issues.apache.org/jira/browse/HDFS-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14510210#comment-14510210
]
Hadoop QA commented on HDFS-6763:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 36s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | javac | 7m 27s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 3m 59s | The applied patch generated 1
additional checkstyle issues. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 7s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | native | 3m 15s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 164m 55s | Tests failed in hadoop-hdfs. |
| | | 209m 29s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.datanode.TestBPOfferService |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12727653/HDFS-6763.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / ef4e996 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/10362/artifact/patchprocess/checkstyle-result-diff.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/10362/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/10362/testReport/ |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10362/console |
This message was automatically generated.
Initialize file system-wide quota once on transitioning to active
-
Key: HDFS-6763
URL: https://issues.apache.org/jira/browse/HDFS-6763
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, namenode
Reporter: Daryn Sharp
Assignee: Kihwal Lee
Attachments: HDFS-6763.patch
{{FSImage#loadEdits}} calls {{updateCountForQuota}} to recalculate verify
quotas for the entire namespace. A standby NN using shared edits calls this
method every minute. The standby may appear to hang for many seconds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8213) DFSClient should not instantiate SpanReceiverHost
[ https://issues.apache.org/jira/browse/HDFS-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508544#comment-14508544 ] Brahma Reddy Battula commented on HDFS-8213: Thanks [~billie.rinaldi] for raising issue and [~cmccabe] for your inputs .. can you people suggest configuration for DFSClient..? DFSClient should not instantiate SpanReceiverHost - Key: HDFS-8213 URL: https://issues.apache.org/jira/browse/HDFS-8213 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.0 Reporter: Billie Rinaldi Assignee: Brahma Reddy Battula Priority: Critical DFSClient initializing SpanReceivers is a problem for Accumulo, which manages SpanReceivers through its own configuration. This results in the same receivers being registered multiple times and spans being delivered more than once. The documentation says SpanReceiverHost.getInstance should be issued once per process, so there is no expectation that DFSClient should do this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8223) Should calculate checksum for parity blocks in DFSStripedOutputStream
[
https://issues.apache.org/jira/browse/HDFS-8223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu updated HDFS-8223:
-
Attachment: HDFS-8223.001.patch
Upload the patch.
The idea is to calculate the checksums for a parity cell once, and then send
them in the packets.
We don't need to add new tests, existing {{TestDFSStripedOutputStream}} can
cover the testing. Since if we calculate checksum for the striped blocks and
write into the packet, then DataNode will check it, I have verified this
through debug too.
I don't use {{byteArrayManager}} to allocate bytes buffer for {{checksumBuf}},
since the buffer is small and only one in one stream instance.
Should calculate checksum for parity blocks in DFSStripedOutputStream
-
Key: HDFS-8223
URL: https://issues.apache.org/jira/browse/HDFS-8223
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-8223.001.patch
{quote}
DFSPacket p = createPacket(packetSize, chunksPerPacket,
streamer.getBytesCurBlock(),
streamer.getAndIncCurrentSeqno(), false);
int maxBytesToPacket = p.getMaxChunks() * bytesPerChecksum;
int toWrite = byteBuffer.remaining() maxBytesToPacket ?
maxBytesToPacket: byteBuffer.remaining();
p.writeData(byteBuffer, toWrite);
streamer.incBytesCurBlock(toWrite);
packets.add(p);
{quote}
In {{DFSStripedOutputStream}}, have not calculated checksum for parity blocks
in {{DFSStripedOutputStream}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[
https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508511#comment-14508511
]
Hadoop QA commented on HDFS-8033:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12727544/hdfs8033-HDFS-7285.04.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 18eb5e7 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10357/console |
This message was automatically generated.
Erasure coding: stateful (non-positional) read from files in striped layout
---
Key: HDFS-8033
URL: https://issues.apache.org/jira/browse/HDFS-8033
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8033.000.patch, HDFS-8033.001.patch,
HDFS-8033.002.patch, HDFS-8033.003.patch, hdfs8033-HDFS-7285.04.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8217) During block recovery for truncate Log new Block Id in case of copy-on-truncate is true.
[
https://issues.apache.org/jira/browse/HDFS-8217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508512#comment-14508512
]
Vinayakumar B commented on HDFS-8217:
-
Thanks [~zero45] for taking a look at the patch.
bq. My only nit, I see you are changing the style of the LOG.info from String
concat to StringBuilder.
Yes I agree that style is changed. But It depends on how you use while building
big messages and *conditional* messages.
In the first patch (.01.patch) if you see, I followed the earlier approach, but
since I used inline conditional statements I got checkstyle comment from
[~hadoopqa].
And I thought StringBuilder would be better.
During block recovery for truncate Log new Block Id in case of
copy-on-truncate is true.
Key: HDFS-8217
URL: https://issues.apache.org/jira/browse/HDFS-8217
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Reporter: Vinayakumar B
Assignee: Vinayakumar B
Attachments: HDFS-8217-01.patch, HDFS-8217-02.patch
During the block recovery for the truncate, include the newblock Id in case
copy-on-truncate set to true.
Current log message is just as below.
{noformat}NameNode at /127.0.0.1:55036 calls
recoverBlock(BP-409377067-xx.xx.xx.253-1429679063755:blk_1073741825_1001,
targets=[DatanodeInfoWithStorage[127.0.0.1:55045,null,null]],
newGenerationStamp=1002){noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[
https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508580#comment-14508580
]
Walter Su commented on HDFS-8033:
-
... ByteBufferStrategy.doRead() ignores len argument. It always read
byteBuffer.remaining, untils EOF of the current block.
Correct myself:
ByteBufferStrategy.doRead() ignores len argument. It always read
byteBuffer.remaining, untils EOF of the current *packet*.
I read {{BlockSender.doSendBlock()}}. I found out that packet size is depended
by io.file.buffer.size and BlockSender.MIN_BUFFER_WITH_TRANSFERTO. If we read
block locally, then size of data part of packet is
io.file.buffer.size(default 4096).
HdfsConstants.BLOCK_STRIPED_CELL_SIZE = 256 * 1024;
Good thing is, cellSize%packetSize == 0, 256 * 1024 /4096 == 4; so we call
{{ByteBufferStrategy.doRead()}} 4 times. We can read exactly one cell.
What if cellSize%packetSize != 0? It'll be wrong.
Try config io.file.buffer.size == 4099. The testcase will failed. ( any other
value cellSize%packetSize != 0 )
Your implementation for bytebuffer works now. But We have to make sure,
cellSize % (io.file.buffer.size) ==0 (for local read)
cellSize % (BlockSender.MIN_BUFFER_WITH_TRANSFERTO) ==0 (for remote read)
*When we choose another value for cellSize , we should be careful. Otherwise
read(bytebuffer) won't work.*
Erasure coding: stateful (non-positional) read from files in striped layout
---
Key: HDFS-8033
URL: https://issues.apache.org/jira/browse/HDFS-8033
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8033.000.patch, HDFS-8033.001.patch,
HDFS-8033.002.patch, HDFS-8033.003.patch, hdfs8033-HDFS-7285.04.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8201) Add an end to end test for stripping file writing and reading
[ https://issues.apache.org/jira/browse/HDFS-8201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinwei Qin updated HDFS-8201: -- Attachment: HDFS-8201.001.patch Add an end to end test for stripping file writing and reading - Key: HDFS-8201 URL: https://issues.apache.org/jira/browse/HDFS-8201 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Xinwei Qin Attachments: HDFS-8201.001.patch According to off-line discussion with [~zhz] and [~xinwei], we need to implement an end to end test for stripping file support: * Create an EC zone; * Create a file in the zone; * Write various typical sizes of content to the file, each size maybe a test method; * Read the written content back; * Compare the written content and read content to ensure it's good; The test facility is subject to add more steps for erasure encoding and recovering. Will open separate issue for it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8231) StackTrace displayed at client while QuotaByStorageType exceeds
[
https://issues.apache.org/jira/browse/HDFS-8231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508993#comment-14508993
]
J.Andreina commented on HDFS-8231:
--
Observation:
==
Following exception is thrown at client side
{noformat}
15/04/23 18:12:07 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.QuotaByStorageTypeExceededException):
Quota by storage type : ARCHIVE on path : /Folder1 is exceeded. quota = 10 B
but space consumed = 384 MB
at
org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyQuotaByStorageType(DirectoryWithQuotaFeature.java:227)
at
org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyQuota(DirectoryWithQuotaFeature.java:240)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:874)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:707)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:666)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:491)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveAllocatedBlock(FSNamesystem.java:3562)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.storeAllocatedBlock(FSNamesystem.java:3140)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3019)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:727)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:489)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2142)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2138)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2136)
at org.apache.hadoop.ipc.Client.call(Client.java:1492)
at org.apache.hadoop.ipc.Client.call(Client.java:1423)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
at com.sun.proxy.$Proxy15.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DataStreamer.locateFollowingBlock(DataStreamer.java:1540)
at
org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1334)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:425)
put: Quota by storage type : ARCHIVE on path : /Folder1 is exceeded. quota = 10
B but space consumed = 384 MB
{noformat}
StackTrace displayed at client while QuotaByStorageType exceeds
---
Key: HDFS-8231
URL: https://issues.apache.org/jira/browse/HDFS-8231
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Reporter: J.Andreina
Assignee: J.Andreina
Priority: Minor
StackTrace displayed at client while QuotaByStorageType exceeds.
With reference to HDFS-2360, feel better to fix this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8231) StackTrace displayed at client while QuotaByStorageType exceeds
J.Andreina created HDFS-8231: Summary: StackTrace displayed at client while QuotaByStorageType exceeds Key: HDFS-8231 URL: https://issues.apache.org/jira/browse/HDFS-8231 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Reporter: J.Andreina Assignee: J.Andreina Priority: Minor StackTrace displayed at client while QuotaByStorageType exceeds. With reference to HDFS-2360, feel better to fix this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8202) Improve end to end stirpping file test to add erasure recovering test
[ https://issues.apache.org/jira/browse/HDFS-8202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinwei Qin updated HDFS-8202: -- Description: This to follow on HDFS-8201 to add erasure recovering test in the end to end stripping file test: * After writing certain blocks to the test file, delete some block file; * Read the file content and compare, see if any recovering issue, or verify the erasure recovering works or not. was: This to follow on HDFS-8021 to add erasure recovering test in the end to end stripping file test: * After writing certain blocks to the test file, delete some block file; * Read the file content and compare, see if any recovering issue, or verify the erasure recovering works or not. Improve end to end stirpping file test to add erasure recovering test - Key: HDFS-8202 URL: https://issues.apache.org/jira/browse/HDFS-8202 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Xinwei Qin This to follow on HDFS-8201 to add erasure recovering test in the end to end stripping file test: * After writing certain blocks to the test file, delete some block file; * Read the file content and compare, see if any recovering issue, or verify the erasure recovering works or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8200) Refactor FSDirStatAndListingOp
[
https://issues.apache.org/jira/browse/HDFS-8200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508610#comment-14508610
]
Hadoop QA commented on HDFS-8200:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 31s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | javac | 7m 27s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 40s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 6m 50s | The applied patch generated 2
additional checkstyle issues. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 7s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | native | 3m 12s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 171m 29s | Tests failed in hadoop-hdfs. |
| | | 218m 45s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12726982/HDFS-8200.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a100be6 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/10355/artifact/patchprocess/checkstyle-result-diff.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/10355/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/10355/testReport/ |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10355//console |
This message was automatically generated.
Refactor FSDirStatAndListingOp
--
Key: HDFS-8200
URL: https://issues.apache.org/jira/browse/HDFS-8200
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-8200.000.patch, HDFS-8200.001.patch
After HDFS-6826 several functions in {{FSDirStatAndListingOp}} are dead. This
jira proposes to clean them up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-4448) HA NN does not start with wildcard address configured for other NN when security is enabled
[
https://issues.apache.org/jira/browse/HDFS-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508674#comment-14508674
]
Arun Suresh commented on HDFS-4448:
---
No tests were included since as [~atm] had mentioned, Tested this manually on a
4 node secure-cluster and ensured both NNs came up and DNs were able to talk to
them.
HA NN does not start with wildcard address configured for other NN when
security is enabled
---
Key: HDFS-4448
URL: https://issues.apache.org/jira/browse/HDFS-4448
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, namenode, security
Affects Versions: 2.0.3-alpha
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Fix For: 2.8.0
Attachments: HDFS-4448.2.patch, HDFS-4448.patch, HDFS-4448.patch
Currently if one tries to configure HA NNs use the wildcard HTTP address when
security is enabled, the NN will fail to start with an error like the
following:
{code}
java.lang.IllegalArgumentException: java.io.IOException: Cannot use a
wildcard address with security. Must explicitly set bind address for Kerberos
{code}
This is the case even if one configures an actual address for the other NN's
HTTP address. There's no good reason for this, since we now check for the
local address being set to 0.0.0.0 and determine the canonical hostname for
Kerberos purposes using
{{InetAddress.getLocalHost().getCanonicalHostName()}}, so we should remove
the restriction.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-4448) Allow HA NN to start in secure mode with wildcard address configured
[
https://issues.apache.org/jira/browse/HDFS-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HDFS-4448:
--
Summary: Allow HA NN to start in secure mode with wildcard address
configured (was: Allow HA NN to start with wildcard address configured)
Allow HA NN to start in secure mode with wildcard address configured
Key: HDFS-4448
URL: https://issues.apache.org/jira/browse/HDFS-4448
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, namenode, security
Affects Versions: 2.0.3-alpha
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Fix For: 2.8.0
Attachments: HDFS-4448.2.patch, HDFS-4448.patch, HDFS-4448.patch
Currently if one tries to configure HA NNs use the wildcard HTTP address when
security is enabled, the NN will fail to start with an error like the
following:
{code}
java.lang.IllegalArgumentException: java.io.IOException: Cannot use a
wildcard address with security. Must explicitly set bind address for Kerberos
{code}
This is the case even if one configures an actual address for the other NN's
HTTP address. There's no good reason for this, since we now check for the
local address being set to 0.0.0.0 and determine the canonical hostname for
Kerberos purposes using
{{InetAddress.getLocalHost().getCanonicalHostName()}}, so we should remove
the restriction.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8230) Erasure Coding: Ignore DatanodeProtocol#DNA_ERASURE_CODING_RECOVERY commands from standbynode if any
Uma Maheswara Rao G created HDFS-8230: - Summary: Erasure Coding: Ignore DatanodeProtocol#DNA_ERASURE_CODING_RECOVERY commands from standbynode if any Key: HDFS-8230 URL: https://issues.apache.org/jira/browse/HDFS-8230 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Priority: Minor Let's ignore DNA_ERASURE_CODING_RECOVERY command at DN if its coming from standby namenode. Ideally we should not get this commands from standby node. Since we handle to ignore the commands from standby node, we can add this also to be ignored. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HDFS-8203) Erasure Coding: Seek and other Ops in DFSStripedInputStream.
[
https://issues.apache.org/jira/browse/HDFS-8203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-8203 started by Yi Liu.
Erasure Coding: Seek and other Ops in DFSStripedInputStream.
Key: HDFS-8203
URL: https://issues.apache.org/jira/browse/HDFS-8203
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
In HDFS-7782 and HDFS-8033, we handle pread and stateful read for
{{DFSStripedInputStream}}, we also need handle other operations, such as
{{seek}}, zerocopy read ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8229) LAZY_PERSIST be deleted after restart hdfs
surendra singh lilhore created HDFS-8229:
Summary: LAZY_PERSIST be deleted after restart hdfs
Key: HDFS-8229
URL: https://issues.apache.org/jira/browse/HDFS-8229
Project: Hadoop HDFS
Issue Type: Bug
Components: HDFS
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
{code}
2015-04-20 10:26:55,180 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Removing lazyPersist file
/LAZY_PERSIST/smallfile with no replicas.
{code}
After namenode restart and before block reports if {{LazyPersistFileScrubber}}
will run then it will delete Lazy persist file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8227) BlockCollection should return Block instead of BlockInfoContiguous in the interfaces
[
https://issues.apache.org/jira/browse/HDFS-8227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508695#comment-14508695
]
Hadoop QA commented on HDFS-8227:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 1s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 8 new or modified test files. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | javac | 7m 47s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 5s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 5m 34s | The applied patch generated 3
additional checkstyle issues. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 3m 9s | The patch appears to introduce 1
new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | native | 3m 19s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 164m 21s | Tests passed in hadoop-hdfs.
|
| | | 212m 1s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| | Redundant nullcheck of snapshotBlocks, which is known to be non-null in
org.apache.hadoop.hdfs.server.namenode.INodeFile.isBlockInLatestSnapshot(Block)
Redundant null check at INodeFile.java:is known to be non-null in
org.apache.hadoop.hdfs.server.namenode.INodeFile.isBlockInLatestSnapshot(Block)
Redundant null check at INodeFile.java:[line 922] |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12727538/HDFS-8227.000.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 18eb5e7 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/10356/artifact/patchprocess/checkstyle-result-diff.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/10356/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/10356/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/10356/testReport/ |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10356/console |
This message was automatically generated.
BlockCollection should return Block instead of BlockInfoContiguous in the
interfaces
Key: HDFS-8227
URL: https://issues.apache.org/jira/browse/HDFS-8227
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-8227.000.patch
Currently the {{BlockCollection}} interface returns {{BlockInfoContiguous}}
objects in multiple calls.
In order to decouple block manager and the namespace, the interface should
return the {{Block}} instead of {{BlockInfoContiguous}} objects, where the
latter contain information like the locations of DN.
This jira proposes to clean up the interface and to return the {{Block}}
objects directly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-4448) Allow HA NN to start in secure mode with wildcard address configured
[
https://issues.apache.org/jira/browse/HDFS-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508715#comment-14508715
]
Hudson commented on HDFS-4448:
--
FAILURE: Integrated in Hadoop-trunk-Commit #7645 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7645/])
HDFS-4448. Allow HA NN to start in secure mode with wildcard address configured
(atm via asuresh) (Arun Suresh: rev baf8bc6c488de170d2caf76d9fa4c99faaa8f1a6)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
Allow HA NN to start in secure mode with wildcard address configured
Key: HDFS-4448
URL: https://issues.apache.org/jira/browse/HDFS-4448
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, namenode, security
Affects Versions: 2.0.3-alpha
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Fix For: 2.8.0
Attachments: HDFS-4448.2.patch, HDFS-4448.patch, HDFS-4448.patch
Currently if one tries to configure HA NNs use the wildcard HTTP address when
security is enabled, the NN will fail to start with an error like the
following:
{code}
java.lang.IllegalArgumentException: java.io.IOException: Cannot use a
wildcard address with security. Must explicitly set bind address for Kerberos
{code}
This is the case even if one configures an actual address for the other NN's
HTTP address. There's no good reason for this, since we now check for the
local address being set to 0.0.0.0 and determine the canonical hostname for
Kerberos purposes using
{{InetAddress.getLocalHost().getCanonicalHostName()}}, so we should remove
the restriction.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7337) Configurable and pluggable Erasure Codec and schema
[ https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508740#comment-14508740 ] jack liuquan commented on HDFS-7337: Hi Kai, As I know, Hitchhiker code can be configured the same as RS code. Using system defined schemas RS(6,3) and RS(10, 4) is OK. Hitchhikercodec can also be configured as you showing in PluggableErasureCodec-v3.pdf. Configurable and pluggable Erasure Codec and schema --- Key: HDFS-7337 URL: https://issues.apache.org/jira/browse/HDFS-7337 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Kai Zheng Attachments: HDFS-7337-prototype-v1.patch, HDFS-7337-prototype-v2.zip, HDFS-7337-prototype-v3.zip, PluggableErasureCodec-v2.pdf, PluggableErasureCodec-v3.pdf, PluggableErasureCodec.pdf According to HDFS-7285 and the design, this considers to support multiple Erasure Codecs via pluggable approach. It allows to define and configure multiple codec schemas with different coding algorithms and parameters. The resultant codec schemas can be utilized and specified via command tool for different file folders. While design and implement such pluggable framework, it’s also to implement a concrete codec by default (Reed Solomon) to prove the framework is useful and workable. Separate JIRA could be opened for the RS codec implementation. Note HDFS-7353 will focus on the very low level codec API and implementation to make concrete vendor libraries transparent to the upper layer. This JIRA focuses on high level stuffs that interact with configuration, schema and etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8228) testFileSmallerThanOneStripe2 failed
Kai Zheng created HDFS-8228:
---
Summary: testFileSmallerThanOneStripe2 failed
Key: HDFS-8228
URL: https://issues.apache.org/jira/browse/HDFS-8228
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Kai Zheng
Assignee: Li Bo
Fix For: HDFS-7285
Playing with the branch, found below:
{noformat}
java.io.EOFException: Premature EOF: no length prefix available
at
org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2343)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:244)
at
org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:823)
2015-04-23 23:21:08,666 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped
SelectChannelConnector@localhost:0
2015-04-23 23:21:08,769 INFO ipc.Server (Server.java:stop(2540)) - Stopping
server on 57920
2015-04-23 23:21:08,770 INFO datanode.DataNode
(BlockReceiver.java:receiveBlock(826)) - Exception for
BP-1850767374-10.239.12.51-1429802363548:blk_-9223372036854775737_1007
java.io.InterruptedIOException: Interrupted while waiting for IO on channel
java.nio.channels.SocketChannel[closed]. 6 millis timeout left.
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:201)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:472)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:787)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:793)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
at java.lang.Thread.run(Thread.java:745)
2015-04-23 23:21:08,769 INFO datanode.DataNode (BlockReceiver.java:run(1250))
- PacketResponder:
BP-1850767374-10.239.12.51-1429802363548:blk_-9223372036854775737_1007,
type=LAST_IN_PIPELINE, downstreams=0:[]: Thread is interrupted.
2015-04-23 23:21:08,776 WARN datanode.DataNode
(BPServiceActor.java:offerService(756)) - BPOfferService for Block pool
BP-1850767374-10.239.12.51-1429802363548 (Datanode Uuid
72b12e39-77cb-463d-a919-0ac06d166fcd) service to localhost/127.0.0.1:36877
interrupted
2015-04-23 23:21:08,776 INFO ipc.Server (Server.java:run(846)) - Stopping IPC
Server Responder
{noformat}
[~libo-intel]], would you help with this?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-8136) Client gets and uses EC schema when reads and writes a stripping file
[
https://issues.apache.org/jira/browse/HDFS-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng resolved HDFS-8136.
-
Resolution: Fixed
Fix Version/s: HDFS-7285
Hadoop Flags: Reviewed
Client gets and uses EC schema when reads and writes a stripping file
-
Key: HDFS-8136
URL: https://issues.apache.org/jira/browse/HDFS-8136
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Zheng
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8136-005.patch, HDFS-8136.1.patch,
HDFS-8136.2.patch, HDFS-8136.3.patch, HDFS-8136.4.patch, HDFS-8136.6.patch,
HDFS-8136.7.patch
Discussed with [~umamaheswararao] and [~vinayrpet], in client when reading
and writing a stripping file, it can invoke a separate call to NameNode to
request the EC schema associated with the EC zone where the file is in. Then
the schema can be used to guide the reading and writing. Currently it uses
hard-coded values.
Optionally, as an optimization consideration, client may cache schema info
per file or per zone or per schema name. We could add schema name in
{{HdfsFileStatus}} for that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8229) LAZY_PERSIST file gets deleted after restart hdfs
[
https://issues.apache.org/jira/browse/HDFS-8229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
surendra singh lilhore updated HDFS-8229:
-
Summary: LAZY_PERSIST file gets deleted after restart hdfs (was:
LAZY_PERSIST be deleted after restart hdfs)
LAZY_PERSIST file gets deleted after restart hdfs
-
Key: HDFS-8229
URL: https://issues.apache.org/jira/browse/HDFS-8229
Project: Hadoop HDFS
Issue Type: Bug
Components: HDFS
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
{code}
2015-04-20 10:26:55,180 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Removing lazyPersist
file /LAZY_PERSIST/smallfile with no replicas.
{code}
After namenode restart and before block reports if
{{LazyPersistFileScrubber}} will run then it will delete Lazy persist file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-4448) Allow HA NN to start in secure mode with wildcard address configured
[
https://issues.apache.org/jira/browse/HDFS-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HDFS-4448:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
Allow HA NN to start in secure mode with wildcard address configured
Key: HDFS-4448
URL: https://issues.apache.org/jira/browse/HDFS-4448
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, namenode, security
Affects Versions: 2.0.3-alpha
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Fix For: 2.8.0
Attachments: HDFS-4448.2.patch, HDFS-4448.patch, HDFS-4448.patch
Currently if one tries to configure HA NNs use the wildcard HTTP address when
security is enabled, the NN will fail to start with an error like the
following:
{code}
java.lang.IllegalArgumentException: java.io.IOException: Cannot use a
wildcard address with security. Must explicitly set bind address for Kerberos
{code}
This is the case even if one configures an actual address for the other NN's
HTTP address. There's no good reason for this, since we now check for the
local address being set to 0.0.0.0 and determine the canonical hostname for
Kerberos purposes using
{{InetAddress.getLocalHost().getCanonicalHostName()}}, so we should remove
the restriction.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-4448) Allow HA NN to start in secure mode with wildcard address configured
[
https://issues.apache.org/jira/browse/HDFS-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508684#comment-14508684
]
Arun Suresh commented on HDFS-4448:
---
+1
Committed to trunk and branch-2.
Thanks [~atm]
Allow HA NN to start in secure mode with wildcard address configured
Key: HDFS-4448
URL: https://issues.apache.org/jira/browse/HDFS-4448
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, namenode, security
Affects Versions: 2.0.3-alpha
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Fix For: 2.8.0
Attachments: HDFS-4448.2.patch, HDFS-4448.patch, HDFS-4448.patch
Currently if one tries to configure HA NNs use the wildcard HTTP address when
security is enabled, the NN will fail to start with an error like the
following:
{code}
java.lang.IllegalArgumentException: java.io.IOException: Cannot use a
wildcard address with security. Must explicitly set bind address for Kerberos
{code}
This is the case even if one configures an actual address for the other NN's
HTTP address. There's no good reason for this, since we now check for the
local address being set to 0.0.0.0 and determine the canonical hostname for
Kerberos purposes using
{{InetAddress.getLocalHost().getCanonicalHostName()}}, so we should remove
the restriction.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8228) TestDFSStripedInputStream#testFileSmallerThanOneStripe2 failed
[
https://issues.apache.org/jira/browse/HDFS-8228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HDFS-8228:
Summary: TestDFSStripedInputStream#testFileSmallerThanOneStripe2 failed
(was: testFileSmallerThanOneStripe2 failed)
TestDFSStripedInputStream#testFileSmallerThanOneStripe2 failed
--
Key: HDFS-8228
URL: https://issues.apache.org/jira/browse/HDFS-8228
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Kai Zheng
Assignee: Li Bo
Fix For: HDFS-7285
Playing with the branch, found below:
{noformat}
java.io.EOFException: Premature EOF: no length prefix available
at
org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2343)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:244)
at
org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:823)
2015-04-23 23:21:08,666 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped
SelectChannelConnector@localhost:0
2015-04-23 23:21:08,769 INFO ipc.Server (Server.java:stop(2540)) - Stopping
server on 57920
2015-04-23 23:21:08,770 INFO datanode.DataNode
(BlockReceiver.java:receiveBlock(826)) - Exception for
BP-1850767374-10.239.12.51-1429802363548:blk_-9223372036854775737_1007
java.io.InterruptedIOException: Interrupted while waiting for IO on channel
java.nio.channels.SocketChannel[closed]. 6 millis timeout left.
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:201)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:472)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:787)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:793)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:250)
at java.lang.Thread.run(Thread.java:745)
2015-04-23 23:21:08,769 INFO datanode.DataNode
(BlockReceiver.java:run(1250)) - PacketResponder:
BP-1850767374-10.239.12.51-1429802363548:blk_-9223372036854775737_1007,
type=LAST_IN_PIPELINE, downstreams=0:[]: Thread is interrupted.
2015-04-23 23:21:08,776 WARN datanode.DataNode
(BPServiceActor.java:offerService(756)) - BPOfferService for Block pool
BP-1850767374-10.239.12.51-1429802363548 (Datanode Uuid
72b12e39-77cb-463d-a919-0ac06d166fcd) service to localhost/127.0.0.1:36877
interrupted
2015-04-23 23:21:08,776 INFO ipc.Server (Server.java:run(846)) - Stopping
IPC Server Responder
{noformat}
[~libo-intel]], would you help with this?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8136) Client gets and uses EC schema when reads and writes a stripping file
[
https://issues.apache.org/jira/browse/HDFS-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508647#comment-14508647
]
Kai Zheng commented on HDFS-8136:
-
The latest patch looks great! Thanks [~kaisasak]. I will commit it shortly.
Client gets and uses EC schema when reads and writes a stripping file
-
Key: HDFS-8136
URL: https://issues.apache.org/jira/browse/HDFS-8136
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Zheng
Assignee: Kai Sasaki
Attachments: HDFS-8136-005.patch, HDFS-8136.1.patch,
HDFS-8136.2.patch, HDFS-8136.3.patch, HDFS-8136.4.patch, HDFS-8136.6.patch,
HDFS-8136.7.patch
Discussed with [~umamaheswararao] and [~vinayrpet], in client when reading
and writing a stripping file, it can invoke a separate call to NameNode to
request the EC schema associated with the EC zone where the file is in. Then
the schema can be used to guide the reading and writing. Currently it uses
hard-coded values.
Optionally, as an optimization consideration, client may cache schema info
per file or per zone or per schema name. We could add schema name in
{{HdfsFileStatus}} for that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-4448) Allow HA NN to start with wildcard address configured.
[
https://issues.apache.org/jira/browse/HDFS-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HDFS-4448:
--
Summary: Allow HA NN to start with wildcard address configured. (was: HA
NN does not start with wildcard address configured for other NN when security
is enabled)
Allow HA NN to start with wildcard address configured.
--
Key: HDFS-4448
URL: https://issues.apache.org/jira/browse/HDFS-4448
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, namenode, security
Affects Versions: 2.0.3-alpha
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Fix For: 2.8.0
Attachments: HDFS-4448.2.patch, HDFS-4448.patch, HDFS-4448.patch
Currently if one tries to configure HA NNs use the wildcard HTTP address when
security is enabled, the NN will fail to start with an error like the
following:
{code}
java.lang.IllegalArgumentException: java.io.IOException: Cannot use a
wildcard address with security. Must explicitly set bind address for Kerberos
{code}
This is the case even if one configures an actual address for the other NN's
HTTP address. There's no good reason for this, since we now check for the
local address being set to 0.0.0.0 and determine the canonical hostname for
Kerberos purposes using
{{InetAddress.getLocalHost().getCanonicalHostName()}}, so we should remove
the restriction.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-4448) Allow HA NN to start with wildcard address configured
[
https://issues.apache.org/jira/browse/HDFS-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HDFS-4448:
--
Summary: Allow HA NN to start with wildcard address configured (was: Allow
HA NN to start with wildcard address configured.)
Allow HA NN to start with wildcard address configured
-
Key: HDFS-4448
URL: https://issues.apache.org/jira/browse/HDFS-4448
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, namenode, security
Affects Versions: 2.0.3-alpha
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Fix For: 2.8.0
Attachments: HDFS-4448.2.patch, HDFS-4448.patch, HDFS-4448.patch
Currently if one tries to configure HA NNs use the wildcard HTTP address when
security is enabled, the NN will fail to start with an error like the
following:
{code}
java.lang.IllegalArgumentException: java.io.IOException: Cannot use a
wildcard address with security. Must explicitly set bind address for Kerberos
{code}
This is the case even if one configures an actual address for the other NN's
HTTP address. There's no good reason for this, since we now check for the
local address being set to 0.0.0.0 and determine the canonical hostname for
Kerberos purposes using
{{InetAddress.getLocalHost().getCanonicalHostName()}}, so we should remove
the restriction.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8215) Refactor NamenodeFsck#check method
[ https://issues.apache.org/jira/browse/HDFS-8215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-8215: --- Description: This method is very long, approximately 230 lines. Splitting this method into smaller methods. (was: This method is very long, approximately 190 lines. Splitting this method into smaller methods.) Refactor NamenodeFsck#check method -- Key: HDFS-8215 URL: https://issues.apache.org/jira/browse/HDFS-8215 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Takanobu Asanuma Assignee: Takanobu Asanuma This method is very long, approximately 230 lines. Splitting this method into smaller methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
