[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-3689: --- Fix Version/s: (was: 3.0.0) Add support for variable length block - Key: HDFS-3689 URL: https://issues.apache.org/jira/browse/HDFS-3689 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, hdfs-client, namenode Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Jing Zhao Fix For: 2.7.0 Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch, HDFS-3689.009.patch, HDFS-3689.009.patch, HDFS-3689.010.patch, HDFS-3689.branch-2.patch, editsStored Currently HDFS supports fixed length blocks. Supporting variable length block will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Fix Version/s: 2.7.0 Add support for variable length block - Key: HDFS-3689 URL: https://issues.apache.org/jira/browse/HDFS-3689 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, hdfs-client, namenode Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Jing Zhao Fix For: 3.0.0, 2.7.0 Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch, HDFS-3689.009.patch, HDFS-3689.009.patch, HDFS-3689.010.patch, HDFS-3689.branch-2.patch, editsStored Currently HDFS supports fixed length blocks. Supporting variable length block will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[
https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-3689:
Attachment: HDFS-3689.branch-2.patch
Post the patch for branch-2. So far I have not found any place where the
functionality is broken by the variable length block. Maybe we should merge
this to branch-2 this week? Note that the variable length block will not be
generated unless the user explicitly passes in the {{CreateFlag#NEW_BLOCK}}
while creating the file. Also if we find anything is broken by this feature we
can fix them in separate jiras.
Add support for variable length block
-
Key: HDFS-3689
URL: https://issues.apache.org/jira/browse/HDFS-3689
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, hdfs-client, namenode
Affects Versions: 3.0.0
Reporter: Suresh Srinivas
Assignee: Jing Zhao
Fix For: 3.0.0
Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch,
HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch,
HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch,
HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch,
HDFS-3689.009.patch, HDFS-3689.009.patch, HDFS-3689.010.patch,
HDFS-3689.branch-2.patch, editsStored
Currently HDFS supports fixed length blocks. Supporting variable length block
will allow new use cases and features to be built on top of HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[
https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-3689:
Release Note:
1. HDFS now can choose to append data to a new block instead of end of the last
partial block. Users can pass {{CreateFlag.APPEND}} and
{{CreateFlag.NEW_BLOCK}} to the {{append}} API to indicate this requirement.
2. HDFS now allows users to pass {{SyncFlag.END_BLOCK}} to the {{hsync}} API to
finish the current block and write remaining data to a new block.
Add support for variable length block
-
Key: HDFS-3689
URL: https://issues.apache.org/jira/browse/HDFS-3689
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, hdfs-client, namenode
Affects Versions: 3.0.0
Reporter: Suresh Srinivas
Assignee: Jing Zhao
Fix For: 3.0.0
Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch,
HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch,
HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch,
HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch,
HDFS-3689.009.patch, HDFS-3689.009.patch, HDFS-3689.010.patch,
HDFS-3689.branch-2.patch, editsStored
Currently HDFS supports fixed length blocks. Supporting variable length block
will allow new use cases and features to be built on top of HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-3689: -- Hadoop Flags: Reviewed +1 patch looks good. Add support for variable length block - Key: HDFS-3689 URL: https://issues.apache.org/jira/browse/HDFS-3689 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, hdfs-client, namenode Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Jing Zhao Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch, HDFS-3689.009.patch, HDFS-3689.009.patch, HDFS-3689.010.patch, editsStored Currently HDFS supports fixed length blocks. Supporting variable length block will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Thanks for the review, Nicholas! I've committed this to trunk. And thanks to all for the discussion. Add support for variable length block - Key: HDFS-3689 URL: https://issues.apache.org/jira/browse/HDFS-3689 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, hdfs-client, namenode Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Jing Zhao Fix For: 3.0.0 Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch, HDFS-3689.009.patch, HDFS-3689.009.patch, HDFS-3689.010.patch, editsStored Currently HDFS supports fixed length blocks. Supporting variable length block will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[
https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-3689:
Attachment: HDFS-3689.010.patch
Thanks Nicholas! Update the patch to address the comments.
bq. The if-condition below should
This if-condition has actually been covered by {{verifyQuota}}. Thus I only
update the comment here.
Add support for variable length block
-
Key: HDFS-3689
URL: https://issues.apache.org/jira/browse/HDFS-3689
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, hdfs-client, namenode
Affects Versions: 3.0.0
Reporter: Suresh Srinivas
Assignee: Jing Zhao
Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch,
HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch,
HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch,
HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch,
HDFS-3689.009.patch, HDFS-3689.009.patch, HDFS-3689.010.patch, editsStored
Currently HDFS supports fixed length blocks. Supporting variable length block
will allow new use cases and features to be built on top of HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.009.patch Add support for variable length block - Key: HDFS-3689 URL: https://issues.apache.org/jira/browse/HDFS-3689 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, hdfs-client, namenode Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Jing Zhao Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch, HDFS-3689.009.patch, HDFS-3689.009.patch Currently HDFS supports fixed length blocks. Supporting variable length block will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: (was: HDFS-3689.009.patch) Add support for variable length block - Key: HDFS-3689 URL: https://issues.apache.org/jira/browse/HDFS-3689 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, hdfs-client, namenode Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Jing Zhao Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch, HDFS-3689.009.patch Currently HDFS supports fixed length blocks. Supporting variable length block will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.009.patch editsStored Add support for variable length block - Key: HDFS-3689 URL: https://issues.apache.org/jira/browse/HDFS-3689 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, hdfs-client, namenode Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Jing Zhao Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch, HDFS-3689.009.patch, HDFS-3689.009.patch, editsStored Currently HDFS supports fixed length blocks. Supporting variable length block will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.009.patch Thanks again Nicholas! Update the patch to add quota verification. Add support for variable length block - Key: HDFS-3689 URL: https://issues.apache.org/jira/browse/HDFS-3689 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, hdfs-client, namenode Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Jing Zhao Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch, HDFS-3689.009.patch Currently HDFS supports fixed length blocks. Supporting variable length block will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[
https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-3689:
Attachment: HDFS-3689.008.patch
Thanks for the comments, Colin. To use CreateFlag is a good suggestion. Update
the patch to address your comments.
Currently the patch targets for 3.0. We can merge this into 2.x after making
sure this feature does not break existing applications' functionalities. So far
I only checked {{FileInputFormat}} and looks like variable length block may
only affect its performance but will not break its functionality.
Add support for variable length block
-
Key: HDFS-3689
URL: https://issues.apache.org/jira/browse/HDFS-3689
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, hdfs-client, namenode
Affects Versions: 3.0.0
Reporter: Suresh Srinivas
Assignee: Jing Zhao
Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch,
HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch,
HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch,
HDFS-3689.007.patch, HDFS-3689.008.patch
Currently HDFS supports fixed length blocks. Supporting variable length block
will allow new use cases and features to be built on top of HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.008.patch Add support for variable length block - Key: HDFS-3689 URL: https://issues.apache.org/jira/browse/HDFS-3689 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, hdfs-client, namenode Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Jing Zhao Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, HDFS-3689.007.patch, HDFS-3689.008.patch, HDFS-3689.008.patch Currently HDFS supports fixed length blocks. Supporting variable length block will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.007.patch Thanks for the review, Nicholas! Update the patch to address the comments. bq. We also need to enforce the same replication in concat if we don't want to update disk quota. The new patch just updates the diskspace quota usage after the concat. Add support for variable length block - Key: HDFS-3689 URL: https://issues.apache.org/jira/browse/HDFS-3689 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, hdfs-client, namenode Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Jing Zhao Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch, HDFS-3689.007.patch Currently HDFS supports fixed length blocks. Supporting variable length block will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[
https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-3689:
Attachment: HDFS-3689.006.patch
Thanks for the review, Nicholas! Update the patch to address the comments.
Here's a summary of the changes:
# remove the new Append2 API and add a boolean {{newBlock}} to
{{ClientProtocol#append}} to indicate whether the data should be appended to a
new block
# still add a new OP_APPEND editlog and use it also for the old append
operation (instead of using OP_ADD)
# update the APPEND inotify event
# Fix the bug in flushOrSync as pointed by Nicholas
Add support for variable length block
-
Key: HDFS-3689
URL: https://issues.apache.org/jira/browse/HDFS-3689
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, hdfs-client, namenode
Affects Versions: 3.0.0
Reporter: Suresh Srinivas
Assignee: Jing Zhao
Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch,
HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch,
HDFS-3689.004.patch, HDFS-3689.005.patch, HDFS-3689.006.patch
Currently HDFS supports fixed length blocks. Supporting variable length block
will allow new use cases and features to be built on top of HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.004.patch Add support for variable length block - Key: HDFS-3689 URL: https://issues.apache.org/jira/browse/HDFS-3689 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, hdfs-client, namenode Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Jing Zhao Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, HDFS-3689.004.patch Currently HDFS supports fixed length blocks. Supporting variable length block will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.005.patch After an offline discussion with [~sanjay.radia] and [~szetszwo], the 005 patch still keeps the restriction that the source files and the target file should be in the same directory. In this way we do not need to update the quota. Add support for variable length block - Key: HDFS-3689 URL: https://issues.apache.org/jira/browse/HDFS-3689 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, hdfs-client, namenode Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Jing Zhao Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch, HDFS-3689.004.patch, HDFS-3689.005.patch Currently HDFS supports fixed length blocks. Supporting variable length block will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.003.patch Add support for variable length block - Key: HDFS-3689 URL: https://issues.apache.org/jira/browse/HDFS-3689 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, hdfs-client, namenode Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Jing Zhao Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, HDFS-3689.002.patch, HDFS-3689.003.patch, HDFS-3689.003.patch Currently HDFS supports fixed length blocks. Supporting variable length block will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.003.patch Update the patch with concat support. Add support for variable length block - Key: HDFS-3689 URL: https://issues.apache.org/jira/browse/HDFS-3689 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, hdfs-client, namenode Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Jing Zhao Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, HDFS-3689.002.patch, HDFS-3689.003.patch Currently HDFS supports fixed length blocks. Supporting variable length block will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Assignee: Jing Zhao (was: Suresh Srinivas) Status: Patch Available (was: Open) Add support for variable length block - Key: HDFS-3689 URL: https://issues.apache.org/jira/browse/HDFS-3689 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, hdfs-client, namenode Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Jing Zhao Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, HDFS-3689.002.patch Currently HDFS supports fixed length blocks. Supporting variable length block will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.002.patch Rebase the patch. Add support for variable length block - Key: HDFS-3689 URL: https://issues.apache.org/jira/browse/HDFS-3689 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, hdfs-client, namenode Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, HDFS-3689.002.patch Currently HDFS supports fixed length blocks. Supporting variable length block will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[
https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-3689:
Attachment: HDFS-3689.001.patch
Update the patch a little bit to demonstrate #3: the support in DFSOutputStream
to let clients specify when to allocate a new block. Currently I simply add a
new SyncFlag named END_BLOCK. Clients can call {{hsync(END_BLOCK)}} to force
the completion of the current block. Some unit tests are also included to make
sure the basic functionality works.
Add support for variable length block
-
Key: HDFS-3689
URL: https://issues.apache.org/jira/browse/HDFS-3689
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, hdfs-client, namenode
Affects Versions: 3.0.0
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch
Currently HDFS supports fixed length blocks. Supporting variable length block
will allow new use cases and features to be built on top of HDFS.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3689: Attachment: HDFS-3689.000.patch Upload a preliminary patch that addresses the above #3 and #4. Specifically a new append RPC is added which always appends new data to a new block, and thus the previous last block becomes a block with variable length. Some unit tests are added to make sure the appended file can still be read/pread by the current DFSInputStream. If this is a correct direction towards variable block length support, the remaining work can be: # bug fix and code cleanup # loose the restriction of concat # add support in DFSOutputStream to let clients specify when to allocate a new block Add support for variable length block - Key: HDFS-3689 URL: https://issues.apache.org/jira/browse/HDFS-3689 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, hdfs-client, namenode Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: HDFS-3689.000.patch Currently HDFS supports fixed length blocks. Supporting variable length block will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
