[jira] [Commented] (HDFS-7116) Add a metric to expose the bandwidth of balancer
[ https://issues.apache.org/jira/browse/HDFS-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708464#comment-14708464 ] Allen Wittenauer commented on HDFS-7116: Could we please stop adding camel case options to command lines? Add a metric to expose the bandwidth of balancer Key: HDFS-7116 URL: https://issues.apache.org/jira/browse/HDFS-7116 Project: Hadoop HDFS Issue Type: New Feature Components: balancer mover Reporter: Akira AJISAKA Assignee: Rakesh R Attachments: HDFS-7116-00.patch, HDFS-7116-01.patch, HDFS-7116-02.patch, HDFS-7116-03.patch, HDFS-7116-04.patch, HDFS-7116-05.patch Now reading logs is the only way to check how the balancer bandwidth is set. It would be useful for administrators if they can get the value of the same. This jira to discuss implement the way to access the balancer bandwidth value of the datanode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7116) Add a metric to expose the bandwidth of balancer
[ https://issues.apache.org/jira/browse/HDFS-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708546#comment-14708546 ] Tsz Wo Nicholas Sze commented on HDFS-7116: --- Command options are case insensitive. Add a metric to expose the bandwidth of balancer Key: HDFS-7116 URL: https://issues.apache.org/jira/browse/HDFS-7116 Project: Hadoop HDFS Issue Type: New Feature Components: balancer mover Reporter: Akira AJISAKA Assignee: Rakesh R Attachments: HDFS-7116-00.patch, HDFS-7116-01.patch, HDFS-7116-02.patch, HDFS-7116-03.patch, HDFS-7116-04.patch, HDFS-7116-05.patch Now reading logs is the only way to check how the balancer bandwidth is set. It would be useful for administrators if they can get the value of the same. This jira to discuss implement the way to access the balancer bandwidth value of the datanode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8931) Erasure Coding: Notify exception to client side from ParityGenerator.
[
https://issues.apache.org/jira/browse/HDFS-8931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Sasaki updated HDFS-8931:
-
Summary: Erasure Coding: Notify exception to client side from
ParityGenerator. (was: Notify exception to client side from ParityGenerator.)
Erasure Coding: Notify exception to client side from ParityGenerator.
-
Key: HDFS-8931
URL: https://issues.apache.org/jira/browse/HDFS-8931
Project: Hadoop HDFS
Issue Type: Sub-task
Components: HDFS
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Labels: EC
Fix For: HDFS-7285
Following HDFS-8287.
Current client thread catch up the exception from {{ParityGenerator}}. In
order to handle properly,
1. Put together handling logic into UncaughtExceptionHandler.
2. Notify exception to client side from UncaughtExceptionHandler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7285) Erasure Coding Support inside HDFS
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] GAO Rui updated HDFS-7285: -- Attachment: HDFSErasureCodingSystemTestPlan-20150824.pdf Inspired by [^HDFSErasureCodingPhaseITestPlan.pdf] created by [~zhz]. We has implemented system tests(Functional Tests) according to this test plan [^HDFSErasureCodingSystemTestPlan-20150824.pdf]. Latest test report will be posted separately soon.Please feel free to discuss and add more test cases or scenarios to this system test plan. Erasure Coding Support inside HDFS -- Key: HDFS-7285 URL: https://issues.apache.org/jira/browse/HDFS-7285 Project: Hadoop HDFS Issue Type: New Feature Reporter: Weihua Jiang Assignee: Zhe Zhang Attachments: Consolidated-20150707.patch, Consolidated-20150806.patch, Consolidated-20150810.patch, ECAnalyzer.py, ECParser.py, HDFS-7285-initial-PoC.patch, HDFS-7285-merge-consolidated-01.patch, HDFS-7285-merge-consolidated-trunk-01.patch, HDFS-7285-merge-consolidated.trunk.03.patch, HDFS-7285-merge-consolidated.trunk.04.patch, HDFS-EC-Merge-PoC-20150624.patch, HDFS-EC-merge-consolidated-01.patch, HDFS-bistriped.patch, HDFSErasureCodingDesign-20141028.pdf, HDFSErasureCodingDesign-20141217.pdf, HDFSErasureCodingDesign-20150204.pdf, HDFSErasureCodingDesign-20150206.pdf, HDFSErasureCodingPhaseITestPlan.pdf, HDFSErasureCodingSystemTestPlan-20150824.pdf, fsimage-analysis-20150105.pdf Erasure Coding (EC) can greatly reduce the storage overhead without sacrifice of data reliability, comparing to the existing HDFS 3-replica approach. For example, if we use a 10+4 Reed Solomon coding, we can allow loss of 4 blocks, with storage overhead only being 40%. This makes EC a quite attractive alternative for big data storage, particularly for cold data. Facebook had a related open source project called HDFS-RAID. It used to be one of the contribute packages in HDFS but had been removed since Hadoop 2.0 for maintain reason. The drawbacks are: 1) it is on top of HDFS and depends on MapReduce to do encoding and decoding tasks; 2) it can only be used for cold files that are intended not to be appended anymore; 3) the pure Java EC coding implementation is extremely slow in practical use. Due to these, it might not be a good idea to just bring HDFS-RAID back. We (Intel and Cloudera) are working on a design to build EC into HDFS that gets rid of any external dependencies, makes it self-contained and independently maintained. This design lays the EC feature on the storage type support and considers compatible with existing HDFS features like caching, snapshot, encryption, high availability and etc. This design will also support different EC coding schemes, implementations and policies for different deployment scenarios. By utilizing advanced libraries (e.g. Intel ISA-L library), an implementation can greatly improve the performance of EC encoding/decoding and makes the EC solution even more attractive. We will post the design document soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8763) Incremental blockreport order may replicate unnecessary block
[
https://issues.apache.org/jira/browse/HDFS-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708726#comment-14708726
]
Walter Su commented on HDFS-8763:
-
The description gives good logs but not a good reason. It's a race condition
when you close the small file too early. After file closed, a race between IBR
of 3rd replica and replication scheduler.
You have closed the file before x.x.7.73 node reports the 3rd finalized replica
of lastBlock. So the lastBlock is scheduled to replicate. After x.x.7.73
reported the 3rd replica. The lastBlock should be removed from
{{needReplications}}. But if the replication command is sent to a new
DN(x.x.4.65 node). There is no way to cancel the command from x.x.4.65 node.
The solution is to postpone scheduling replication, and wait x.x.7.73 to report
the 3rd replica.
Incremental blockreport order may replicate unnecessary block
-
Key: HDFS-8763
URL: https://issues.apache.org/jira/browse/HDFS-8763
Project: Hadoop HDFS
Issue Type: Bug
Components: HDFS
Affects Versions: 2.4.0
Reporter: jiangyu
Priority: Minor
For our cluster, the NameNode is always very busy, so for every incremental
block report , the contention of lock is heavy.
The logic of incremental block report is as follow, client send block to dn1
and dn1 mirror to dn2, dn2 mirror to dn3. After finish this block, all
datanode will report the newly received block to namenode. In NameNode side,
all will go to the method processIncrementalBlockReport in BlockManager
class. But the status of the block reported from dn2,dn3 is RECEIVING_BLOCK,
for dn1 is RECEIED_BLOCK. It is okay if dn2, dn3 report before dn1(that is
common), but in some busy environment, it is easy to find dn1 report before
dn2 or dn3, let’s assume dn2 report first, dn1 report second, and dn3 report
third.
So dn1 will addStoredBlock and find the replica of this block is not reach
the the original number(which is 3), and the block will add to
neededReplications construction and soon ask some node in pipeline (dn1 or
dn2)to replica it dn4 . After sometime, dn4 and dn3 all report this block,
then choose one node to invalidate.
Here is one log i found in our cluster:
2015-07-08 01:05:34,675 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocateBlock:
/logs/***_bigdata_spam/logs/application_1435099124107_470749/xx.xx.4.62_45454.tmp.
BP-1386326728-xx.xx.2.131-1382089338395
blk_3194502674_2121080184{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[[DISK]DS-a7c0f8f6-2399-4980-9479-efa08487b7b3:NORMAL|RBW],
ReplicaUnderConstruction[[DISK]DS-c75145a0-ed63-4180-87ee-d48ccaa647c5:NORMAL|RBW],
ReplicaUnderConstruction[[DISK]DS-15a4dc8e-5b7d-449f-a941-6dced45e6f07:NORMAL|RBW]]}
2015-07-08 01:05:34,689 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: xx.xx.7.75:50010 is added to
blk_3194502674_2121080184{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[[DISK]DS-15a4dc8e-5b7d-449f-a941-6dced45e6f07:NORMAL|RBW],
ReplicaUnderConstruction[[DISK]DS-74ed264f-da43-4cc3-9fa9-164ba99f752a:NORMAL|RBW],
ReplicaUnderConstruction[[DISK]DS-56121ce1-8991-45b3-95bc-2a5357991512:NORMAL|RBW]]}
size 0
2015-07-08 01:05:34,689 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: xx.xx.4.62:50010 is added to
blk_3194502674_2121080184{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[[DISK]DS-15a4dc8e-5b7d-449f-a941-6dced45e6f07:NORMAL|RBW],
ReplicaUnderConstruction[[DISK]DS-74ed264f-da43-4cc3-9fa9-164ba99f752a:NORMAL|RBW],
ReplicaUnderConstruction[[DISK]DS-56121ce1-8991-45b3-95bc-2a5357991512:NORMAL|RBW]]}
size 0
2015-07-08 01:05:35,003 INFO BlockStateChange: BLOCK* ask xx.xx.4.62:50010 to
replicate blk_3194502674_2121080184 to datanode(s) xx.xx.4.65:50010
2015-07-08 01:05:35,403 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: xx.xx.7.73:50010 is added to blk_3194502674_2121080184 size
67750
2015-07-08 01:05:35,833 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: xx.xx.4.65:50010 is added to blk_3194502674_2121080184 size
67750
2015-07-08 01:05:35,833 INFO BlockStateChange: BLOCK* InvalidateBlocks: add
blk_3194502674_2121080184 to xx.xx.7.75:50010
2015-07-08 01:05:35,833 INFO BlockStateChange: BLOCK* chooseExcessReplicates:
(xx.xx.7.75:50010, blk_3194502674_2121080184) is added to invalidated blocks
set
2015-07-08 01:05:35,852 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
InvalidateBlocks: ask xx.xx.7.75:50010 to delete [blk_3194502674_2121080184,
blk_3194497594_2121075104]
Some day, the number of this situation can be 40, that is not good for
[jira] [Commented] (HDFS-8388) Time and Date format need to be in sync in Namenode UI page
[
https://issues.apache.org/jira/browse/HDFS-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708784#comment-14708784
]
Vinayakumar B commented on HDFS-8388:
-
1. {code} var HELPERS = {
'helper_date_tostring' : function (chunk, ctx, bodies, params) {
var value = dust.helpers.tap(params.value, chunk, ctx);
-return chunk.write('' + new Date(Number(value)).toLocaleString());
+return chunk.write('' + moment(Number(v)).format('ddd MMM DD HH:mm:ss
ZZ '));
}
};{code}
Here, {{Number(v)}}, there is no {{v}}. You need to replace with {{value}}.
2. Need to fix 2 checkstyles out of 3 shown
3. You can clean lines with whitespace endings, though this can be done during
commit.
In addition to the above changes, just remove duplicate line in
{{dfshealth.html}}. Its not from this issue, but i think that also can go along
with this patch.
{{/scriptscript type=text/javascript src=/static/moment.min.js}}
Time and Date format need to be in sync in Namenode UI page
---
Key: HDFS-8388
URL: https://issues.apache.org/jira/browse/HDFS-8388
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Archana T
Assignee: Surendra Singh Lilhore
Priority: Minor
Attachments: HDFS-8388-002.patch, HDFS-8388-003.patch,
HDFS-8388-004.patch, HDFS-8388-005.patch, HDFS-8388.patch, HDFS-8388_1.patch,
ScreenShot-InvalidDate.png
In NameNode UI Page, Date and Time FORMAT displayed on the page are not in
sync currently.
Started:Wed May 13 12:28:02 IST 2015
Compiled:23 Apr 2015 12:22:59
Block Deletion Start Time 13 May 2015 12:28:02
We can keep a common format in all the above places.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8943) Read apis in ByteRangeInputStream does not read all the bytes specified when chunked transfer-encoding is used in the server
[
https://issues.apache.org/jira/browse/HDFS-8943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shradha Revankar updated HDFS-8943:
---
Description:
With the default Webhdfs server implementation the read apis in
ByteRangeInputStream work as expected reading the correct number of bytes for
these apis :
{{public int read(byte b[], int off, int len)}}
{{public int read(long position, byte[] buffer, int offset, int length)}}
But when a custom Webhdfs server implementation is plugged in which uses
chunked Transfer-encoding, these apis read only the first chunk. Simple fix
would be to loop and read till bytes specified similar to {{readfully()}}
was:
With the default Webhdfs server implementation the read apis in
ByteRangeInputStream work as expected reading the correct number of bytes for
these apis :
{{public int read(long position, byte[] buffer, int offset, int length)}}
{{public int read(long position, byte[] buffer, int offset, int length)}}
But when a custom Webhdfs server implementation is plugged in which uses
chunked Transfer-encoding, these apis read only the first chunk. Simple fix
would be to loop and read till bytes specified similar to {{readfully()}}
Read apis in ByteRangeInputStream does not read all the bytes specified when
chunked transfer-encoding is used in the server
Key: HDFS-8943
URL: https://issues.apache.org/jira/browse/HDFS-8943
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.7.1
Reporter: Shradha Revankar
Assignee: Shradha Revankar
With the default Webhdfs server implementation the read apis in
ByteRangeInputStream work as expected reading the correct number of bytes for
these apis :
{{public int read(byte b[], int off, int len)}}
{{public int read(long position, byte[] buffer, int offset, int length)}}
But when a custom Webhdfs server implementation is plugged in which uses
chunked Transfer-encoding, these apis read only the first chunk. Simple fix
would be to loop and read till bytes specified similar to {{readfully()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8943) Read apis in ByteRangeInputStream does not read all the bytes specified when chunked transfer-encoding is used in the server
[
https://issues.apache.org/jira/browse/HDFS-8943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shradha Revankar updated HDFS-8943:
---
Description:
With the default Webhdfs server implementation the read apis in
ByteRangeInputStream work as expected reading the correct number of bytes for
these apis :
{{public int read(long position, byte[] buffer, int offset, int length) }}
{{public int read(long position, byte[] buffer, int offset, int length)}}
But when a custom Webhdfs server implementation is plugged in which uses
chunked Transfer-encoding, these apis read only the first chunk. Simple fix
would be to loop and read till bytes specified similar to {{readfully()}}
was:
With the default Webhdfs server implementation the read apis in
ByteRangeInputStream work as expected reading the correct number of bytes for
these apis :
{{ public int read(long position, byte[] buffer, int offset, int length)
throws IOException }}
{{public int read(long position, byte[] buffer, int offset, int length)}}
But when a custom Webhdfs server implementation is plugged in which uses
chunked Transfer-encoding, these apis read only the first chunk. Simple fix
would be to loop and read till bytes specified similar to {{readfully()}}
Read apis in ByteRangeInputStream does not read all the bytes specified when
chunked transfer-encoding is used in the server
Key: HDFS-8943
URL: https://issues.apache.org/jira/browse/HDFS-8943
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.7.1
Reporter: Shradha Revankar
Assignee: Shradha Revankar
With the default Webhdfs server implementation the read apis in
ByteRangeInputStream work as expected reading the correct number of bytes for
these apis :
{{public int read(long position, byte[] buffer, int offset, int length) }}
{{public int read(long position, byte[] buffer, int offset, int length)}}
But when a custom Webhdfs server implementation is plugged in which uses
chunked Transfer-encoding, these apis read only the first chunk. Simple fix
would be to loop and read till bytes specified similar to {{readfully()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8943) Read apis in ByteRangeInputStream does not read all the bytes specified when chunked transfer-encoding is used in the server
[
https://issues.apache.org/jira/browse/HDFS-8943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shradha Revankar updated HDFS-8943:
---
Description:
With the default Webhdfs server implementation the read apis in
ByteRangeInputStream work as expected reading the correct number of bytes for
these apis :
{{public int read(long position, byte[] buffer, int offset, int length)}}
{{public int read(long position, byte[] buffer, int offset, int length)}}
But when a custom Webhdfs server implementation is plugged in which uses
chunked Transfer-encoding, these apis read only the first chunk. Simple fix
would be to loop and read till bytes specified similar to {{readfully()}}
was:
With the default Webhdfs server implementation the read apis in
ByteRangeInputStream work as expected reading the correct number of bytes for
these apis :
{{public int read(long position, byte[] buffer, int offset, int length) }}
{{public int read(long position, byte[] buffer, int offset, int length)}}
But when a custom Webhdfs server implementation is plugged in which uses
chunked Transfer-encoding, these apis read only the first chunk. Simple fix
would be to loop and read till bytes specified similar to {{readfully()}}
Read apis in ByteRangeInputStream does not read all the bytes specified when
chunked transfer-encoding is used in the server
Key: HDFS-8943
URL: https://issues.apache.org/jira/browse/HDFS-8943
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.7.1
Reporter: Shradha Revankar
Assignee: Shradha Revankar
With the default Webhdfs server implementation the read apis in
ByteRangeInputStream work as expected reading the correct number of bytes for
these apis :
{{public int read(long position, byte[] buffer, int offset, int length)}}
{{public int read(long position, byte[] buffer, int offset, int length)}}
But when a custom Webhdfs server implementation is plugged in which uses
chunked Transfer-encoding, these apis read only the first chunk. Simple fix
would be to loop and read till bytes specified similar to {{readfully()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8942) Update hyperlink to rack awareness page in HDFS Architecture documentation
[ https://issues.apache.org/jira/browse/HDFS-8942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-8942: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) I've committed this to trunk and branch-2. Thanks [~iwasakims] for the contribution. Update hyperlink to rack awareness page in HDFS Architecture documentation -- Key: HDFS-8942 URL: https://issues.apache.org/jira/browse/HDFS-8942 Project: Hadoop HDFS Issue Type: Bug Components: documentation Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Trivial Fix For: 2.8.0 Attachments: HDFS-8942.001.patch The hyperlink to rack awareness documentation in HdfsDesign.md is pointing to non-existent target. It should be fixed to point to RackAwareness.html. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8944) Make dfsadmin command options case insensitive
[
https://issues.apache.org/jira/browse/HDFS-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708788#comment-14708788
]
Rakesh R commented on HDFS-8944:
Thanks [~ajisakaa] for the quick updates. With this change we are removing the
restriction of case sensitiveness for the command options. IMHO, its good to
discuss about backward compatibility part and update {{Release Note:}} field in
the jira.
Make dfsadmin command options case insensitive
--
Key: HDFS-8944
URL: https://issues.apache.org/jira/browse/HDFS-8944
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Minor
Attachments: HDFS-8944.001.patch
Now dfsadmin command options are case sensitive except allowSnapshot and
disallowSnapshot. It would be better to make them case insensitive for
usability and consistency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8900) Compact XAttrs to optimize memory footprint.
[
https://issues.apache.org/jira/browse/HDFS-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu updated HDFS-8900:
-
Attachment: HDFS-8900.003.patch
Compact XAttrs to optimize memory footprint.
Key: HDFS-8900
URL: https://issues.apache.org/jira/browse/HDFS-8900
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-8900.001.patch, HDFS-8900.002.patch,
HDFS-8900.003.patch
{code}
private final ImmutableListXAttr xAttrs;
{code}
Currently we use above in XAttrFeature, it's not efficient from memory point
of view, since {{ImmutableList}} and {{XAttr}} have object memory overhead,
and each object has memory alignment.
We can use a {{byte[]}} in XAttrFeature and do some compact in {{XAttr}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8935) Erasure Coding: createErasureCodingZone api should accept the policyname as argument instead of ErasureCodingPolicy
[ https://issues.apache.org/jira/browse/HDFS-8935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8935: Parent Issue: HDFS-8031 (was: HDFS-7285) Erasure Coding: createErasureCodingZone api should accept the policyname as argument instead of ErasureCodingPolicy --- Key: HDFS-8935 URL: https://issues.apache.org/jira/browse/HDFS-8935 Project: Hadoop HDFS Issue Type: Sub-task Reporter: J.Andreina Assignee: J.Andreina Current behavior : User has to specify ErasureCodingPolicy as an argument for createErasureCodingZone api . This can be made in sync with creation of EC zone through CLI , where user need to specify only the policy name. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8935) Erasure Coding: createErasureCodingZone api should accept the policyname as argument instead of ErasureCodingPolicy
[ https://issues.apache.org/jira/browse/HDFS-8935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708822#comment-14708822 ] Zhe Zhang commented on HDFS-8935: - I agree with Vinay. Moving it under HDFS-8031 since multi-policy support belongs to follow-on work. Erasure Coding: createErasureCodingZone api should accept the policyname as argument instead of ErasureCodingPolicy --- Key: HDFS-8935 URL: https://issues.apache.org/jira/browse/HDFS-8935 Project: Hadoop HDFS Issue Type: Sub-task Reporter: J.Andreina Assignee: J.Andreina Current behavior : User has to specify ErasureCodingPolicy as an argument for createErasureCodingZone api . This can be made in sync with creation of EC zone through CLI , where user need to specify only the policy name. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8287) DFSStripedOutputStream.writeChunk should not wait for writing parity
[ https://issues.apache.org/jira/browse/HDFS-8287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708691#comment-14708691 ] Kai Sasaki commented on HDFS-8287: -- [~rakeshr] Thank you for reviewing! Can other someone check this JIRA's patch? And I created follow on work of this (HDFS-8931). DFSStripedOutputStream.writeChunk should not wait for writing parity - Key: HDFS-8287 URL: https://issues.apache.org/jira/browse/HDFS-8287 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Reporter: Tsz Wo Nicholas Sze Assignee: Kai Sasaki Attachments: HDFS-8287-HDFS-7285.00.patch, HDFS-8287-HDFS-7285.01.patch, HDFS-8287-HDFS-7285.02.patch, HDFS-8287-HDFS-7285.03.patch, HDFS-8287-HDFS-7285.04.patch, HDFS-8287-HDFS-7285.05.patch When a stripping cell is full, writeChunk computes and generates parity packets. It sequentially calls waitAndQueuePacket so that user client cannot continue to write data until it finishes. We should allow user client to continue writing instead but not blocking it when writing parity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7116) Add a metric to expose the bandwidth of balancer
[
https://issues.apache.org/jira/browse/HDFS-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708696#comment-14708696
]
Akira AJISAKA commented on HDFS-7116:
-
bq. This seems wrong to show it in metric since it is a mostly constant value.
Just like that we won't show other conf values in metric.
I understand. Thanks [~szetszwo] for the comment.
bq. Now, your approach is avoiding this problem. I'll start preparing a patch
if others agree to this approach.
Agree.
bq. Command options are case insensitive.
{code:title=DFSAdmin.java}
try {
if (-report.equals(cmd)) {
report(argv, i);
} else if (-safemode.equals(cmd)) {
setSafeMode(argv, i);
} else if (-allowSnapshot.equalsIgnoreCase(cmd)) {
allowSnapshot(argv);
...
{code}
dfsadmin command options are case sensitive except allowSnapshot and
disallowSnapshot. I'm thinking we should make them case insensitive in a
separate jira.
Add a metric to expose the bandwidth of balancer
Key: HDFS-7116
URL: https://issues.apache.org/jira/browse/HDFS-7116
Project: Hadoop HDFS
Issue Type: New Feature
Components: balancer mover
Reporter: Akira AJISAKA
Assignee: Rakesh R
Attachments: HDFS-7116-00.patch, HDFS-7116-01.patch,
HDFS-7116-02.patch, HDFS-7116-03.patch, HDFS-7116-04.patch, HDFS-7116-05.patch
Now reading logs is the only way to check how the balancer bandwidth is set.
It would be useful for administrators if they can get the value of the same.
This jira to discuss implement the way to access the balancer bandwidth
value of the datanode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8943) Read apis in ByteRangeInputStream does not read all the bytes specified when chunked transfer-encoding is used in the server
Shradha Revankar created HDFS-8943:
--
Summary: Read apis in ByteRangeInputStream does not read all the
bytes specified when chunked transfer-encoding is used in the server
Key: HDFS-8943
URL: https://issues.apache.org/jira/browse/HDFS-8943
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.7.1
Reporter: Shradha Revankar
Assignee: Shradha Revankar
With the default Webhdfs server implementation the read apis in
ByteRangeInputStream work as expected reading the correct number of bytes for
these apis :
{{public int read(long position, byte[] buffer, int offset, int length)
throws IOException }}
{{public int read(long position, byte[] buffer, int offset, int length)}}
But when a custom Webhdfs server implementation is plugged in which uses
chunked Transfer-encoding, these apis read only the first chunk. Simple fix
would be to loop and read till bytes specified similar to {{readfully()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8943) Read apis in ByteRangeInputStream does not read all the bytes specified when chunked transfer-encoding is used in the server
[
https://issues.apache.org/jira/browse/HDFS-8943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shradha Revankar updated HDFS-8943:
---
Description:
With the default Webhdfs server implementation the read apis in
ByteRangeInputStream work as expected reading the correct number of bytes for
these apis :
{{ public int read(long position, byte[] buffer, int offset, int length)
throws IOException }}
{{public int read(long position, byte[] buffer, int offset, int length)}}
But when a custom Webhdfs server implementation is plugged in which uses
chunked Transfer-encoding, these apis read only the first chunk. Simple fix
would be to loop and read till bytes specified similar to {{readfully()}}
was:
With the default Webhdfs server implementation the read apis in
ByteRangeInputStream work as expected reading the correct number of bytes for
these apis :
{{public int read(long position, byte[] buffer, int offset, int length)
throws IOException }}
{{public int read(long position, byte[] buffer, int offset, int length)}}
But when a custom Webhdfs server implementation is plugged in which uses
chunked Transfer-encoding, these apis read only the first chunk. Simple fix
would be to loop and read till bytes specified similar to {{readfully()}}
Read apis in ByteRangeInputStream does not read all the bytes specified when
chunked transfer-encoding is used in the server
Key: HDFS-8943
URL: https://issues.apache.org/jira/browse/HDFS-8943
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.7.1
Reporter: Shradha Revankar
Assignee: Shradha Revankar
With the default Webhdfs server implementation the read apis in
ByteRangeInputStream work as expected reading the correct number of bytes for
these apis :
{{ public int read(long position, byte[] buffer, int offset, int length)
throws IOException }}
{{public int read(long position, byte[] buffer, int offset, int length)}}
But when a custom Webhdfs server implementation is plugged in which uses
chunked Transfer-encoding, these apis read only the first chunk. Simple fix
would be to loop and read till bytes specified similar to {{readfully()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8944) Make dfsadmin command options case insensitive
[ https://issues.apache.org/jira/browse/HDFS-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-8944: Attachment: HDFS-8944.001.patch Attaching a patch that replaces String.equals with String.equalsIgnorecase. Make dfsadmin command options case insensitive -- Key: HDFS-8944 URL: https://issues.apache.org/jira/browse/HDFS-8944 Project: Hadoop HDFS Issue Type: Improvement Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Attachments: HDFS-8944.001.patch Now dfsadmin command options are case sensitive except allowSnapshot and disallowSnapshot. It would be better to make them case insensitive for usability and consistency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8944) Make dfsadmin command options case insensitive
[ https://issues.apache.org/jira/browse/HDFS-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-8944: Target Version/s: 2.8.0 Status: Patch Available (was: Open) Make dfsadmin command options case insensitive -- Key: HDFS-8944 URL: https://issues.apache.org/jira/browse/HDFS-8944 Project: Hadoop HDFS Issue Type: Improvement Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Attachments: HDFS-8944.001.patch Now dfsadmin command options are case sensitive except allowSnapshot and disallowSnapshot. It would be better to make them case insensitive for usability and consistency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7116) Add a metric to expose the bandwidth of balancer
[ https://issues.apache.org/jira/browse/HDFS-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708762#comment-14708762 ] Akira AJISAKA commented on HDFS-7116: - Filed HDFS-8944 for make them case insensitive. Add a metric to expose the bandwidth of balancer Key: HDFS-7116 URL: https://issues.apache.org/jira/browse/HDFS-7116 Project: Hadoop HDFS Issue Type: New Feature Components: balancer mover Reporter: Akira AJISAKA Assignee: Rakesh R Attachments: HDFS-7116-00.patch, HDFS-7116-01.patch, HDFS-7116-02.patch, HDFS-7116-03.patch, HDFS-7116-04.patch, HDFS-7116-05.patch Now reading logs is the only way to check how the balancer bandwidth is set. It would be useful for administrators if they can get the value of the same. This jira to discuss implement the way to access the balancer bandwidth value of the datanode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8942) Update hyperlink to rack awareness page in HDFS Architecture documentation
[ https://issues.apache.org/jira/browse/HDFS-8942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708779#comment-14708779 ] Hudson commented on HDFS-8942: -- FAILURE: Integrated in Hadoop-trunk-Commit #8342 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8342/]) HDFS-8942. Update hyperlink to rack awareness page in HDFS Architecture documentation. Contributed by Masatake Iwasaki. (aajisaka: rev bcaf83902aa4d1e3e2cd26442df0a253eae7f633) * hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Update hyperlink to rack awareness page in HDFS Architecture documentation -- Key: HDFS-8942 URL: https://issues.apache.org/jira/browse/HDFS-8942 Project: Hadoop HDFS Issue Type: Bug Components: documentation Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Trivial Fix For: 2.8.0 Attachments: HDFS-8942.001.patch The hyperlink to rack awareness documentation in HdfsDesign.md is pointing to non-existent target. It should be fixed to point to RackAwareness.html. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8944) Make dfsadmin command options case insensitive
[ https://issues.apache.org/jira/browse/HDFS-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-8944: Priority: Major (was: Minor) Release Note: Now hdfs dfsadmin command options are case insensitive. For example, both hdfs dfsadmin -saveNamespace and hdfs dfsadmin -savenamespace are accepted. Make dfsadmin command options case insensitive -- Key: HDFS-8944 URL: https://issues.apache.org/jira/browse/HDFS-8944 Project: Hadoop HDFS Issue Type: Improvement Reporter: Akira AJISAKA Assignee: Akira AJISAKA Attachments: HDFS-8944.001.patch Now dfsadmin command options are case sensitive except allowSnapshot and disallowSnapshot. It would be better to make them case insensitive for usability and consistency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8942) Update hyperlink to rack awareness page in HDFS Architecture documentation
[ https://issues.apache.org/jira/browse/HDFS-8942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708803#comment-14708803 ] Hudson commented on HDFS-8942: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #298 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/298/]) HDFS-8942. Update hyperlink to rack awareness page in HDFS Architecture documentation. Contributed by Masatake Iwasaki. (aajisaka: rev bcaf83902aa4d1e3e2cd26442df0a253eae7f633) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md Update hyperlink to rack awareness page in HDFS Architecture documentation -- Key: HDFS-8942 URL: https://issues.apache.org/jira/browse/HDFS-8942 Project: Hadoop HDFS Issue Type: Bug Components: documentation Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Trivial Fix For: 2.8.0 Attachments: HDFS-8942.001.patch The hyperlink to rack awareness documentation in HdfsDesign.md is pointing to non-existent target. It should be fixed to point to RackAwareness.html. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8944) Make dfsadmin command options case insensitive
[ https://issues.apache.org/jira/browse/HDFS-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-8944: Release Note: Now hdfs dfsadmin command options are case insensitive. For example, both hdfs dfsadmin -saveNamespace and hdfs dfsadmin -savenamespace are valid. (was: Now hdfs dfsadmin command options are case insensitive. For example, both hdfs dfsadmin -saveNamespace and hdfs dfsadmin -savenamespace are accepted.) Make dfsadmin command options case insensitive -- Key: HDFS-8944 URL: https://issues.apache.org/jira/browse/HDFS-8944 Project: Hadoop HDFS Issue Type: Improvement Reporter: Akira AJISAKA Assignee: Akira AJISAKA Attachments: HDFS-8944.001.patch Now dfsadmin command options are case sensitive except allowSnapshot and disallowSnapshot. It would be better to make them case insensitive for usability and consistency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8944) Make dfsadmin command options case insensitive
[ https://issues.apache.org/jira/browse/HDFS-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708801#comment-14708801 ] Akira AJISAKA commented on HDFS-8944: - Thanks [~rakeshr] for the comment. Updated the release note. I don't think this issue breaks backward compatibility because camel case is supported after the patch, however, this issue and HADOOP-10901 (Thanks Kengo for the link!) should be discussed. Obviously I'm +1 for make them case insensitive for usability. Make dfsadmin command options case insensitive -- Key: HDFS-8944 URL: https://issues.apache.org/jira/browse/HDFS-8944 Project: Hadoop HDFS Issue Type: Improvement Reporter: Akira AJISAKA Assignee: Akira AJISAKA Attachments: HDFS-8944.001.patch Now dfsadmin command options are case sensitive except allowSnapshot and disallowSnapshot. It would be better to make them case insensitive for usability and consistency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8942) Update hyperlink to rack awareness page in HDFS Architecture documentation
[ https://issues.apache.org/jira/browse/HDFS-8942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708817#comment-14708817 ] Hudson commented on HDFS-8942: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #302 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/302/]) HDFS-8942. Update hyperlink to rack awareness page in HDFS Architecture documentation. Contributed by Masatake Iwasaki. (aajisaka: rev bcaf83902aa4d1e3e2cd26442df0a253eae7f633) * hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Update hyperlink to rack awareness page in HDFS Architecture documentation -- Key: HDFS-8942 URL: https://issues.apache.org/jira/browse/HDFS-8942 Project: Hadoop HDFS Issue Type: Bug Components: documentation Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Trivial Fix For: 2.8.0 Attachments: HDFS-8942.001.patch The hyperlink to rack awareness documentation in HdfsDesign.md is pointing to non-existent target. It should be fixed to point to RackAwareness.html. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8900) Compact XAttrs to optimize memory footprint.
[
https://issues.apache.org/jira/browse/HDFS-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708819#comment-14708819
]
Yi Liu commented on HDFS-8900:
--
Thanks [~andrew.wang] and [~cmccabe]!
I will create a follow-on JIRA targeted on trunk to remove the max size config.
Failure of {{TestStartup.testXattrConfiguration}} is related, the new patch
fixes it, and also fixes the checkstyle and whitespace. The findbugs actually
don't exist.
Compact XAttrs to optimize memory footprint.
Key: HDFS-8900
URL: https://issues.apache.org/jira/browse/HDFS-8900
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-8900.001.patch, HDFS-8900.002.patch
{code}
private final ImmutableListXAttr xAttrs;
{code}
Currently we use above in XAttrFeature, it's not efficient from memory point
of view, since {{ImmutableList}} and {{XAttr}} have object memory overhead,
and each object has memory alignment.
We can use a {{byte[]}} in XAttrFeature and do some compact in {{XAttr}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8942) Update hyperlink to rack awareness page in HDFS Architecture documentation
[ https://issues.apache.org/jira/browse/HDFS-8942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-8942: --- Status: Patch Available (was: Open) Update hyperlink to rack awareness page in HDFS Architecture documentation -- Key: HDFS-8942 URL: https://issues.apache.org/jira/browse/HDFS-8942 Project: Hadoop HDFS Issue Type: Bug Components: documentation Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Trivial Attachments: HDFS-8942.001.patch The hyperlink to rack awareness documentation in HdfsDesign.md is pointing to non-existent target. It should be fixed to point to RackAwareness.html. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8763) Incremental blockreport order may replicate unnecessary block
[
https://issues.apache.org/jira/browse/HDFS-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708732#comment-14708732
]
Walter Su commented on HDFS-8763:
-
Another solution is to increase {{dfs.namenode.replication.interval}} to slow
down checker/scheduler.
Incremental blockreport order may replicate unnecessary block
-
Key: HDFS-8763
URL: https://issues.apache.org/jira/browse/HDFS-8763
Project: Hadoop HDFS
Issue Type: Bug
Components: HDFS
Affects Versions: 2.4.0
Reporter: jiangyu
Priority: Minor
For our cluster, the NameNode is always very busy, so for every incremental
block report , the contention of lock is heavy.
The logic of incremental block report is as follow, client send block to dn1
and dn1 mirror to dn2, dn2 mirror to dn3. After finish this block, all
datanode will report the newly received block to namenode. In NameNode side,
all will go to the method processIncrementalBlockReport in BlockManager
class. But the status of the block reported from dn2,dn3 is RECEIVING_BLOCK,
for dn1 is RECEIED_BLOCK. It is okay if dn2, dn3 report before dn1(that is
common), but in some busy environment, it is easy to find dn1 report before
dn2 or dn3, let’s assume dn2 report first, dn1 report second, and dn3 report
third.
So dn1 will addStoredBlock and find the replica of this block is not reach
the the original number(which is 3), and the block will add to
neededReplications construction and soon ask some node in pipeline (dn1 or
dn2)to replica it dn4 . After sometime, dn4 and dn3 all report this block,
then choose one node to invalidate.
Here is one log i found in our cluster:
2015-07-08 01:05:34,675 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocateBlock:
/logs/***_bigdata_spam/logs/application_1435099124107_470749/xx.xx.4.62_45454.tmp.
BP-1386326728-xx.xx.2.131-1382089338395
blk_3194502674_2121080184{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[[DISK]DS-a7c0f8f6-2399-4980-9479-efa08487b7b3:NORMAL|RBW],
ReplicaUnderConstruction[[DISK]DS-c75145a0-ed63-4180-87ee-d48ccaa647c5:NORMAL|RBW],
ReplicaUnderConstruction[[DISK]DS-15a4dc8e-5b7d-449f-a941-6dced45e6f07:NORMAL|RBW]]}
2015-07-08 01:05:34,689 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: xx.xx.7.75:50010 is added to
blk_3194502674_2121080184{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[[DISK]DS-15a4dc8e-5b7d-449f-a941-6dced45e6f07:NORMAL|RBW],
ReplicaUnderConstruction[[DISK]DS-74ed264f-da43-4cc3-9fa9-164ba99f752a:NORMAL|RBW],
ReplicaUnderConstruction[[DISK]DS-56121ce1-8991-45b3-95bc-2a5357991512:NORMAL|RBW]]}
size 0
2015-07-08 01:05:34,689 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: xx.xx.4.62:50010 is added to
blk_3194502674_2121080184{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[[DISK]DS-15a4dc8e-5b7d-449f-a941-6dced45e6f07:NORMAL|RBW],
ReplicaUnderConstruction[[DISK]DS-74ed264f-da43-4cc3-9fa9-164ba99f752a:NORMAL|RBW],
ReplicaUnderConstruction[[DISK]DS-56121ce1-8991-45b3-95bc-2a5357991512:NORMAL|RBW]]}
size 0
2015-07-08 01:05:35,003 INFO BlockStateChange: BLOCK* ask xx.xx.4.62:50010 to
replicate blk_3194502674_2121080184 to datanode(s) xx.xx.4.65:50010
2015-07-08 01:05:35,403 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: xx.xx.7.73:50010 is added to blk_3194502674_2121080184 size
67750
2015-07-08 01:05:35,833 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: xx.xx.4.65:50010 is added to blk_3194502674_2121080184 size
67750
2015-07-08 01:05:35,833 INFO BlockStateChange: BLOCK* InvalidateBlocks: add
blk_3194502674_2121080184 to xx.xx.7.75:50010
2015-07-08 01:05:35,833 INFO BlockStateChange: BLOCK* chooseExcessReplicates:
(xx.xx.7.75:50010, blk_3194502674_2121080184) is added to invalidated blocks
set
2015-07-08 01:05:35,852 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
InvalidateBlocks: ask xx.xx.7.75:50010 to delete [blk_3194502674_2121080184,
blk_3194497594_2121075104]
Some day, the number of this situation can be 40, that is not good for
the performance and waste network band.
Our base version is hadoop 2.4 and i have checked hadoop 2.7.1 didn’t find
any difference.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8944) Make dfsadmin command options case insensitive
[ https://issues.apache.org/jira/browse/HDFS-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-8944: Assignee: Akira AJISAKA Summary: Make dfsadmin command options case insensitive (was: Make dfsadmin command option case insensitive) Make dfsadmin command options case insensitive -- Key: HDFS-8944 URL: https://issues.apache.org/jira/browse/HDFS-8944 Project: Hadoop HDFS Issue Type: Improvement Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Now dfsadmin command options are case sensitive except allowSnapshot and disallowSnapshot. It would be better to make them case insensitive for usability and consistency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8944) Make dfsadmin command option case insensitive
Akira AJISAKA created HDFS-8944: --- Summary: Make dfsadmin command option case insensitive Key: HDFS-8944 URL: https://issues.apache.org/jira/browse/HDFS-8944 Project: Hadoop HDFS Issue Type: Improvement Reporter: Akira AJISAKA Priority: Minor Now dfsadmin command options are case sensitive except allowSnapshot and disallowSnapshot. It would be better to make them case insensitive for usability and consistency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8942) Update hyperlink to rack awareness page in HDFS Architecture documentation
[ https://issues.apache.org/jira/browse/HDFS-8942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-8942: --- Attachment: HDFS-8942.001.patch Update hyperlink to rack awareness page in HDFS Architecture documentation -- Key: HDFS-8942 URL: https://issues.apache.org/jira/browse/HDFS-8942 Project: Hadoop HDFS Issue Type: Bug Components: documentation Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Trivial Attachments: HDFS-8942.001.patch The hyperlink to rack awareness documentation in HdfsDesign.md is pointing to non-existent target. It should be fixed to point to RackAwareness.html. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8942) Update hyperlink to rack awareness page in HDFS Architecture documentation
Masatake Iwasaki created HDFS-8942: -- Summary: Update hyperlink to rack awareness page in HDFS Architecture documentation Key: HDFS-8942 URL: https://issues.apache.org/jira/browse/HDFS-8942 Project: Hadoop HDFS Issue Type: Bug Components: documentation Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Trivial The hyperlink to rack awareness documentation in HdfsDesign.md is pointing to non-existent target. It should be fixed to point to RackAwareness.html. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8942) Update hyperlink to rack awareness page in HDFS Architecture documentation
[ https://issues.apache.org/jira/browse/HDFS-8942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708775#comment-14708775 ] Akira AJISAKA commented on HDFS-8942: - LGTM, +1. Update hyperlink to rack awareness page in HDFS Architecture documentation -- Key: HDFS-8942 URL: https://issues.apache.org/jira/browse/HDFS-8942 Project: Hadoop HDFS Issue Type: Bug Components: documentation Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Trivial Attachments: HDFS-8942.001.patch The hyperlink to rack awareness documentation in HdfsDesign.md is pointing to non-existent target. It should be fixed to point to RackAwareness.html. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
