kangpinghuang commented on a change in pull request #1238: Reduce memory
RowBlock needed
URL: https://github.com/apache/incubator-doris/pull/1238#discussion_r290095012
##########
File path: be/src/olap/row_block.cpp
##########
@@ -314,51 +115,29 @@ void RowBlock::clear() {
}
void RowBlock::_compute_layout() {
+ std::unordered_set<uint32_t> column_set(_info.column_ids.begin(),
_info.column_ids.end());
size_t memory_size = 0;
- size_t storage_fixed_bytes = 0;
- size_t storage_variable_bytes = 0;
- for (auto& field : _tablet_schema) {
+ for (int i = 0; i < _tablet_schema.size(); ++i) {
+ auto& field = _tablet_schema[i];
+ if (!column_set.empty() && column_set.find(i) == std::end(column_set))
{
+ // which may lead BE crash
+
_field_offset_in_memory.push_back(std::numeric_limits<std::size_t>::max());
+ continue;
+ }
+
_field_offset_in_memory.push_back(memory_size);
// All field has a nullbyte in memory
- if (field.type == OLAP_FIELD_TYPE_VARCHAR || field.type ==
OLAP_FIELD_TYPE_HLL) {
+ if (field.type == OLAP_FIELD_TYPE_VARCHAR || field.type ==
OLAP_FIELD_TYPE_HLL
+ || field.type == OLAP_FIELD_TYPE_CHAR) {
// 变长部分额外计算下实际最大的字符串长度(此处length已经包括记录Length的2个字节)
- storage_fixed_bytes += sizeof(StringOffsetType);
- storage_variable_bytes += field.length;
- if (has_nullbyte()) {
- storage_variable_bytes += sizeof(char);
- }
memory_size += sizeof(Slice) + sizeof(char);
} else {
- storage_fixed_bytes += field.length;
- if (has_nullbyte()) {
- storage_fixed_bytes += sizeof(char);
- }
- if (field.type == OLAP_FIELD_TYPE_CHAR) {
- memory_size += sizeof(Slice) + sizeof(char);
- } else {
- memory_size += field.length + sizeof(char);
- }
+ memory_size += field.length + sizeof(char);
Review comment:
we use a char to indicate a null flag, can we use bitmap to realize it in
memory?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]