[jira] [Updated] (HDFS-8580) Erasure coding: Persist cellSize in BlockInfoStriped and StripedBlockProto
[
https://issues.apache.org/jira/browse/HDFS-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Walter Su updated HDFS-8580:
Attachment: HDFS-8580-HDFS-7285.01.patch
Thank [~jingzhao] for ideas. I agree persist it in fsimage.proto. Uploaded 01
patch.
Erasure coding: Persist cellSize in BlockInfoStriped and StripedBlockProto
--
Key: HDFS-8580
URL: https://issues.apache.org/jira/browse/HDFS-8580
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Walter Su
Assignee: Walter Su
Attachments: HDFS-8580-HDFS-7285.01.patch, HDFS-8580.00.patch
Zhe Zhang, Kai Zheng and I had a offline discussion. Here is what we thought:
Add a cellSize field in BlockInfoStriped as a workaround, and deal with
memory usage in follow-on.(HDFS-8059)
discussion in HDFS-8494:
from Walter Su:
{quote}
I think BlockInfoStriped needs to keep cellSize.
{quote}
from [~vinayrpet]:
{quote}
I too was thinking the same when the FSImageLoader problem has came up. This
will increase the memory usage by ~4bytes for each block though.
{quote}
from [~jingzhao]
{quote}
Also, we should consider adding a chunk size field to StripedBlockProto and
removing the cell size field from HdfsFileStatus. In this way we can access
the chunk size information in the storage layer.
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8580) Erasure coding: Persist cellSize in BlockInfoStriped and StripedBlockProto
[
https://issues.apache.org/jira/browse/HDFS-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Walter Su updated HDFS-8580:
Description:
Zhe Zhang, Kai Zheng and I had a offline discussion. Here is what we thought:
Add a cellSize field in BlockInfoStriped as a workaround, and deal with memory
usage in follow-on.(HDFS-8059)
discussion in HDFS-8494:
from Walter Su:
{quote}
I think BlockInfoStriped needs to keep cellSize.
{quote}
from [~vinayrpet]:
{quote}
I too was thinking the same when the FSImageLoader problem has came up.
{quote}
from [~jingzhao]
{quote}
Also, we should consider adding a chunk size field to StripedBlockProto and
removing the cell size field from HdfsFileStatus. In this way we can access the
chunk size information in the storage layer.
{quote}
was:
Zhe Zhang, Kai Zheng and I had a offline discussion. Here is what we thought:
Add a cellSize field in BlockInfoStriped as a workaround, and deal with memory
usage in follow-on.
discussion in HDFS-8494:
from [~jingzhao]
{quote}
Also, we should consider adding a chunk size field to StripedBlockProto and
removing the cell size field from HdfsFileStatus. In this way we can access the
chunk size information in the storage layer.
{quote}
Erasure coding: Persist cellSize in BlockInfoStriped and StripedBlockProto
--
Key: HDFS-8580
URL: https://issues.apache.org/jira/browse/HDFS-8580
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Walter Su
Assignee: Walter Su
Zhe Zhang, Kai Zheng and I had a offline discussion. Here is what we thought:
Add a cellSize field in BlockInfoStriped as a workaround, and deal with
memory usage in follow-on.(HDFS-8059)
discussion in HDFS-8494:
from Walter Su:
{quote}
I think BlockInfoStriped needs to keep cellSize.
{quote}
from [~vinayrpet]:
{quote}
I too was thinking the same when the FSImageLoader problem has came up.
{quote}
from [~jingzhao]
{quote}
Also, we should consider adding a chunk size field to StripedBlockProto and
removing the cell size field from HdfsFileStatus. In this way we can access
the chunk size information in the storage layer.
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8580) Erasure coding: Persist cellSize in BlockInfoStriped and StripedBlockProto
[
https://issues.apache.org/jira/browse/HDFS-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Walter Su updated HDFS-8580:
Description:
Zhe Zhang, Kai Zheng and I had a offline discussion. Here is what we thought:
Add a cellSize field in BlockInfoStriped as a workaround, and deal with memory
usage in follow-on.(HDFS-8059)
discussion in HDFS-8494:
from Walter Su:
{quote}
I think BlockInfoStriped needs to keep cellSize.
{quote}
from [~vinayrpet]:
{quote}
I too was thinking the same when the FSImageLoader problem has came up. This
will increase the memory usage by ~4bytes for each block though.
{quote}
from [~jingzhao]
{quote}
Also, we should consider adding a chunk size field to StripedBlockProto and
removing the cell size field from HdfsFileStatus. In this way we can access the
chunk size information in the storage layer.
{quote}
was:
Zhe Zhang, Kai Zheng and I had a offline discussion. Here is what we thought:
Add a cellSize field in BlockInfoStriped as a workaround, and deal with memory
usage in follow-on.(HDFS-8059)
discussion in HDFS-8494:
from Walter Su:
{quote}
I think BlockInfoStriped needs to keep cellSize.
{quote}
from [~vinayrpet]:
{quote}
I too was thinking the same when the FSImageLoader problem has came up.
{quote}
from [~jingzhao]
{quote}
Also, we should consider adding a chunk size field to StripedBlockProto and
removing the cell size field from HdfsFileStatus. In this way we can access the
chunk size information in the storage layer.
{quote}
Erasure coding: Persist cellSize in BlockInfoStriped and StripedBlockProto
--
Key: HDFS-8580
URL: https://issues.apache.org/jira/browse/HDFS-8580
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Walter Su
Assignee: Walter Su
Zhe Zhang, Kai Zheng and I had a offline discussion. Here is what we thought:
Add a cellSize field in BlockInfoStriped as a workaround, and deal with
memory usage in follow-on.(HDFS-8059)
discussion in HDFS-8494:
from Walter Su:
{quote}
I think BlockInfoStriped needs to keep cellSize.
{quote}
from [~vinayrpet]:
{quote}
I too was thinking the same when the FSImageLoader problem has came up. This
will increase the memory usage by ~4bytes for each block though.
{quote}
from [~jingzhao]
{quote}
Also, we should consider adding a chunk size field to StripedBlockProto and
removing the cell size field from HdfsFileStatus. In this way we can access
the chunk size information in the storage layer.
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8580) Erasure coding: Persist cellSize in BlockInfoStriped and StripedBlockProto
[
https://issues.apache.org/jira/browse/HDFS-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Walter Su updated HDFS-8580:
Attachment: HDFS-8580.00.patch
Erasure coding: Persist cellSize in BlockInfoStriped and StripedBlockProto
--
Key: HDFS-8580
URL: https://issues.apache.org/jira/browse/HDFS-8580
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Walter Su
Assignee: Walter Su
Attachments: HDFS-8580.00.patch
Zhe Zhang, Kai Zheng and I had a offline discussion. Here is what we thought:
Add a cellSize field in BlockInfoStriped as a workaround, and deal with
memory usage in follow-on.(HDFS-8059)
discussion in HDFS-8494:
from Walter Su:
{quote}
I think BlockInfoStriped needs to keep cellSize.
{quote}
from [~vinayrpet]:
{quote}
I too was thinking the same when the FSImageLoader problem has came up. This
will increase the memory usage by ~4bytes for each block though.
{quote}
from [~jingzhao]
{quote}
Also, we should consider adding a chunk size field to StripedBlockProto and
removing the cell size field from HdfsFileStatus. In this way we can access
the chunk size information in the storage layer.
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
