[GitHub] [incubator-doris] gaodayue commented on issue #1684: [Proposal] Bitmap Index File Format for V2 segment
gaodayue commented on issue #1684: [Proposal] Bitmap Index File Format for V2 segment URL: https://github.com/apache/incubator-doris/issues/1684#issuecomment-561043526 @FANLONGFANLONG V2 segment is designed to be immutable. AFAIK, currently Doris implements delete by persisting delete conditions in the metadata and removing deleted rows in the query path. The actual data deletion will happen at compaction time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] lingbin commented on a change in pull request #2359: Improve SkipList memory usage tracking
lingbin commented on a change in pull request #2359: Improve SkipList memory
usage tracking
URL: https://github.com/apache/incubator-doris/pull/2359#discussion_r353021918
##
File path: be/src/olap/memtable.cpp
##
@@ -61,26 +62,49 @@ int MemTable::RowCursorComparator::operator()(const char*
left, const char* righ
return compare_row(lhs_row, rhs_row);
}
-size_t MemTable::memory_usage() {
-return _mem_pool->mem_tracker()->consumption();
-}
+void MemTable::insert(const Tuple* tuple) {
+bool overwritten = false;
+if (_keys_type == KeysType::DUP_KEYS) {
+// Will insert directly, so use memory from _table_mem_pool
+_tuple_buf = _table_mem_pool->allocate(_schema_size);
+ContiguousRow row(_schema, _tuple_buf);
+_tuple_to_row(tuple, , _table_mem_pool.get());
+_skip_list->Insert((char*)_tuple_buf, );
+DCHECK(!overwritten) << "Duplicate key model meet overwrite in
SkipList";
+return;
+}
-void MemTable::insert(Tuple* tuple) {
+_tuple_buf = _tmp_mem_pool->allocate(_schema_size);
ContiguousRow row(_schema, _tuple_buf);
+_tuple_to_row(tuple, , _table_mem_pool.get());
+
+// TODO(lingbin): Remove redundant contain check
+if (_skip_list->Contains((char*)_tuple_buf)) {
+// Will aggregate, use memory from _tmp_mem_pool
+_skip_list->Insert((char*)_tuple_buf, );
+DCHECK(overwritten) << "Does not meet duplicated key in SkipList";
+} else {
+// Will insert directly, so use memory from _table_mem_pool
+_tuple_buf = _table_mem_pool->allocate(_schema_size);
+ContiguousRow dst_row(_schema, _tuple_buf);
+copy_row(_row, row, _table_mem_pool.get());
+_skip_list->Insert((char*)_tuple_buf, );
+DCHECK(!overwritten) << "Meet unexpected duplicated key in SkipList";
+}
+// Make MemPool to be reusable, but does not free its memory
+_tmp_mem_pool->clear();
+}
+
+void MemTable::_tuple_to_row(const Tuple* tuple, ContiguousRow* row, MemPool*
mem_pool) {
Review comment:
My mistake, I will fix it
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2359: Improve SkipList memory usage tracking
imay commented on a change in pull request #2359: Improve SkipList memory usage
tracking
URL: https://github.com/apache/incubator-doris/pull/2359#discussion_r353021046
##
File path: be/src/olap/memtable.cpp
##
@@ -61,26 +62,49 @@ int MemTable::RowCursorComparator::operator()(const char*
left, const char* righ
return compare_row(lhs_row, rhs_row);
}
-size_t MemTable::memory_usage() {
-return _mem_pool->mem_tracker()->consumption();
-}
+void MemTable::insert(const Tuple* tuple) {
+bool overwritten = false;
+if (_keys_type == KeysType::DUP_KEYS) {
+// Will insert directly, so use memory from _table_mem_pool
+_tuple_buf = _table_mem_pool->allocate(_schema_size);
+ContiguousRow row(_schema, _tuple_buf);
+_tuple_to_row(tuple, , _table_mem_pool.get());
+_skip_list->Insert((char*)_tuple_buf, );
+DCHECK(!overwritten) << "Duplicate key model meet overwrite in
SkipList";
+return;
+}
-void MemTable::insert(Tuple* tuple) {
+_tuple_buf = _tmp_mem_pool->allocate(_schema_size);
ContiguousRow row(_schema, _tuple_buf);
+_tuple_to_row(tuple, , _table_mem_pool.get());
+
+// TODO(lingbin): Remove redundant contain check
+if (_skip_list->Contains((char*)_tuple_buf)) {
+// Will aggregate, use memory from _tmp_mem_pool
+_skip_list->Insert((char*)_tuple_buf, );
+DCHECK(overwritten) << "Does not meet duplicated key in SkipList";
+} else {
+// Will insert directly, so use memory from _table_mem_pool
+_tuple_buf = _table_mem_pool->allocate(_schema_size);
+ContiguousRow dst_row(_schema, _tuple_buf);
+copy_row(_row, row, _table_mem_pool.get());
+_skip_list->Insert((char*)_tuple_buf, );
+DCHECK(!overwritten) << "Meet unexpected duplicated key in SkipList";
+}
+// Make MemPool to be reusable, but does not free its memory
+_tmp_mem_pool->clear();
+}
+
+void MemTable::_tuple_to_row(const Tuple* tuple, ContiguousRow* row, MemPool*
mem_pool) {
Review comment:
Seems that input mem_pool is not used.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2360: Support subquery with non-scalar result in Binary predicate and Between-and predicate
imay commented on a change in pull request #2360: Support subquery with
non-scalar result in Binary predicate and Between-and predicate
URL: https://github.com/apache/incubator-doris/pull/2360#discussion_r353005531
##
File path: be/src/exec/assert_num_rows_node.cpp
##
@@ -0,0 +1,76 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/assert_num_rows_node.h"
+
+#include "runtime/row_batch.h"
+#include "runtime/runtime_state.h"
+#include "util/runtime_profile.h"
+#include "gen_cpp/PlanNodes_types.h"
+
+namespace doris {
+
+AssertNumRowsNode::AssertNumRowsNode(
+ObjectPool* pool, const TPlanNode& tnode, const DescriptorTbl& descs) :
+ExecNode(pool, tnode, descs),
+_desired_num_rows(tnode.assert_num_rows_node.desired_num_rows),
+_subquery_string(tnode.assert_num_rows_node.subquery_string) {
+}
+
+Status AssertNumRowsNode::init(const TPlanNode& tnode, RuntimeState* state) {
+RETURN_IF_ERROR(ExecNode::init(tnode, state));
+return Status::OK();
+}
+
+Status AssertNumRowsNode::prepare(RuntimeState* state) {
+RETURN_IF_ERROR(ExecNode::prepare(state));
+return Status::OK();
+}
+
+Status AssertNumRowsNode::open(RuntimeState* state) {
+SCOPED_TIMER(_runtime_profile->total_time_counter());
+RETURN_IF_ERROR(ExecNode::open(state));
+return Status::OK();
+}
+
+Status AssertNumRowsNode::get_next(RuntimeState* state, RowBatch*
output_batch, bool* eos) {
+RETURN_IF_ERROR(exec_debug_action(TExecNodePhase::GETNEXT));
+SCOPED_TIMER(_runtime_profile->total_time_counter());
+child(0)->get_next(state, output_batch, eos);
+int num_rows_returned_before = _num_rows_returned;
+_num_rows_returned += output_batch->num_rows();
+if (_num_rows_returned > _desired_num_rows) {
+_num_rows_returned = num_rows_returned_before;
+LOG(INFO) << "Expected no more than " << _desired_num_rows << " to be
returned by expression "
+ << _subquery_string;
+std::stringstream ss;
+ss << "Expected no more than " << _desired_num_rows << " to be
returned by expression "
+ << _subquery_string;
+return Status::Cancelled(ss.str());
Review comment:
You can try strings::Substitue()
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2360: Support subquery with non-scalar result in Binary predicate and Between-and predicate
imay commented on a change in pull request #2360: Support subquery with non-scalar result in Binary predicate and Between-and predicate URL: https://github.com/apache/incubator-doris/pull/2360#discussion_r353004411 ## File path: fe/src/main/java/org/apache/doris/analysis/StmtRewriter.java ## @@ -331,9 +356,11 @@ private static boolean mergeExpr(SelectStmt stmt, Expr expr, // to eliminate any chance that column aliases from the parent query could reference // select items from the inline view after the rewrite. List colLabels = Lists.newArrayList(); +// 给subquery中的每个结果列取一个新的别名 Review comment: English first This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2360: Support subquery with non-scalar result in Binary predicate and Between-and predicate
imay commented on a change in pull request #2360: Support subquery with
non-scalar result in Binary predicate and Between-and predicate
URL: https://github.com/apache/incubator-doris/pull/2360#discussion_r353007792
##
File path: fe/src/main/java/org/apache/doris/analysis/QueryStmt.java
##
@@ -419,6 +418,14 @@ public long getOffset() {
return limitElement.getOffset();
}
+public void setAssertNumRowsElement(int desiredNumOfRows) {
+this.assertNumRowsElement = new AssertNumRowsElement(desiredNumOfRows,
toSql());
Review comment:
toSql() is error-prone, maybe cause problem in some corner case. It will be
better if there is original statement. Maybe you can leave a TODO here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2360: Support subquery with non-scalar result in Binary predicate and Between-and predicate
imay commented on a change in pull request #2360: Support subquery with
non-scalar result in Binary predicate and Between-and predicate
URL: https://github.com/apache/incubator-doris/pull/2360#discussion_r353002773
##
File path: gensrc/thrift/PlanNodes.thrift
##
@@ -550,6 +551,11 @@ struct TBackendResourceProfile {
4: optional i64 max_row_buffer_size = 4194304 //TODO chenhao
}
+struct TAssertNumRowsNode {
+1: required i32 desired_num_rows;
+2: required string subquery_string;
Review comment:
better to make these optional, maybe we will change this node's definition
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2360: Support subquery with non-scalar result in Binary predicate and Between-and predicate
imay commented on a change in pull request #2360: Support subquery with
non-scalar result in Binary predicate and Between-and predicate
URL: https://github.com/apache/incubator-doris/pull/2360#discussion_r353005903
##
File path: be/src/exec/assert_num_rows_node.h
##
@@ -0,0 +1,41 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/exec_node.h"
+#include "gen_cpp/PlanNodes_types.h"
+
+namespace doris {
+
+// Node for assert row count:
+// -
+class AssertNumRowsNode : public ExecNode {
+public:
+AssertNumRowsNode(ObjectPool* pool, const TPlanNode& tnode, const
DescriptorTbl& descs);
+virtual ~AssertNumRowsNode() {};
+
+virtual Status init(const TPlanNode& tnode, RuntimeState* state = nullptr);
+virtual Status prepare(RuntimeState* state);
+virtual Status open(RuntimeState* state);
+virtual Status get_next(RuntimeState* state, RowBatch* row_batch, bool*
eos);
+virtual Status close(RuntimeState* state);
+
+private:
+int _desired_num_rows;
Review comment:
```suggestion
int64_t _desired_num_rows;
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen closed issue #2352: Make node info metrics available on all FE node
kangkaisen closed issue #2352: Make node info metrics available on all FE node URL: https://github.com/apache/incubator-doris/issues/2352 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2359: Improve SkipList memory usage tracking
kangkaisen commented on a change in pull request #2359: Improve SkipList memory
usage tracking
URL: https://github.com/apache/incubator-doris/pull/2359#discussion_r353002149
##
File path: be/src/olap/skiplist.h
##
@@ -100,9 +101,11 @@ class SkipList {
// Immutable after construction
Comparator const compare_;
+bool _can_dup;
Review comment:
OK. let's keep `_can_dup `.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] lingbin commented on a change in pull request #2359: Improve SkipList memory usage tracking
lingbin commented on a change in pull request #2359: Improve SkipList memory
usage tracking
URL: https://github.com/apache/incubator-doris/pull/2359#discussion_r352999166
##
File path: be/src/olap/skiplist.h
##
@@ -100,9 +101,11 @@ class SkipList {
// Immutable after construction
Comparator const compare_;
+bool _can_dup;
Review comment:
I agree the `SkipList` should not know the Doris data model, and that is why
I changed the `Insert()`(which is aware of data model previously) and add the
`_can_dup` member.
And I use `_can_dup` here, just to emphasize that this value can only be
true under a `DUP` model. Other than this, I think `_can_dup` and `_is_unique`
are the same, I don’t have a strong opinion.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2343: Add classes related to "tag".
kangkaisen commented on a change in pull request #2343: Add classes related to
"tag".
URL: https://github.com/apache/incubator-doris/pull/2343#discussion_r352997682
##
File path: fe/src/main/java/org/apache/doris/resource/Resource.java
##
@@ -0,0 +1,74 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.resource;
+
+import org.apache.doris.common.io.Writable;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+
+/*
+ * Resource is the collective name of the nodes that provide various service
capabilities in Doris cluster.
+ * Each resource has a unique ID.
+ * A resource may have one or more tags that represent the functional
properties or custom groupings of a resource, etc.
+ * eg:
+ * Backend, Frontend, Broker, RemoteStorage
+ */
+public abstract class Resource implements Writable {
Review comment:
Could we serialize and deserialize new metadata class all by Json ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2359: Improve SkipList memory usage tracking
kangkaisen commented on a change in pull request #2359: Improve SkipList memory
usage tracking
URL: https://github.com/apache/incubator-doris/pull/2359#discussion_r352996474
##
File path: be/src/olap/skiplist.h
##
@@ -100,9 +101,11 @@ class SkipList {
// Immutable after construction
Comparator const compare_;
+bool _can_dup;
Review comment:
_is_unique means the Key in skiplist is whether unique, it's not related
Doris data model.
I think skiplist shouldn't know the Doris data model.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] lingbin commented on a change in pull request #2359: Improve SkipList memory usage tracking
lingbin commented on a change in pull request #2359: Improve SkipList memory
usage tracking
URL: https://github.com/apache/incubator-doris/pull/2359#discussion_r352995730
##
File path: be/src/olap/memtable.h
##
@@ -56,7 +61,10 @@ class MemTable {
RowCursorComparator _row_comparator;
std::unique_ptr _mem_tracker;
-std::unique_ptr _mem_pool;
+// This is a buffer, to hold the memory referenced by the rows that have
not
+// been inserted into the SkipList
+std::unique_ptr _tmp_mem_pool;
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] lingbin commented on a change in pull request #2359: Improve SkipList memory usage tracking
lingbin commented on a change in pull request #2359: Improve SkipList memory
usage tracking
URL: https://github.com/apache/incubator-doris/pull/2359#discussion_r352994589
##
File path: be/src/olap/skiplist.h
##
@@ -100,9 +101,11 @@ class SkipList {
// Immutable after construction
Comparator const compare_;
+bool _can_dup;
MemPool* const _mem_pool;// MemPool used for allocations of nodes
Node* const head_;
+// When value is true, means indicates that duplicate values are allowed.
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] lingbin commented on a change in pull request #2359: Improve SkipList memory usage tracking
lingbin commented on a change in pull request #2359: Improve SkipList memory
usage tracking
URL: https://github.com/apache/incubator-doris/pull/2359#discussion_r352994049
##
File path: be/src/olap/skiplist.h
##
@@ -100,9 +101,11 @@ class SkipList {
// Immutable after construction
Comparator const compare_;
+bool _can_dup;
Review comment:
No. There are three models: `AGG`, `UNIQUE`, `DUP`.
Here, we mainly distinguish `DUP` from the other two(`AGG`and `UNIQUE`).
Because only the `DUP` model, the data rows in the memtable can be duplicated.
And on the other hand, `_is_unique` may make other people mistakenly think
"is unique model", right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] lingbin commented on a change in pull request #2359: Improve SkipList memory usage tracking
lingbin commented on a change in pull request #2359: Improve SkipList memory
usage tracking
URL: https://github.com/apache/incubator-doris/pull/2359#discussion_r352994049
##
File path: be/src/olap/skiplist.h
##
@@ -100,9 +101,11 @@ class SkipList {
// Immutable after construction
Comparator const compare_;
+bool _can_dup;
Review comment:
No. There are three models: `AGG`, `UNIQUE`, `DUP`.
Here, we mainly distinguish `DUP` from the other two(`AGG`and `UNIQUE`).
Because only the `DUP` model, the data rows in the memtable can be duplicated.
And on the other hand, `_is_unique` may make other people mistakenly think
"is unique model?"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2359: Improve SkipList memory usage tracking
kangkaisen commented on a change in pull request #2359: Improve SkipList memory
usage tracking
URL: https://github.com/apache/incubator-doris/pull/2359#discussion_r352983843
##
File path: be/src/olap/skiplist.h
##
@@ -100,9 +101,11 @@ class SkipList {
// Immutable after construction
Comparator const compare_;
+bool _can_dup;
MemPool* const _mem_pool;// MemPool used for allocations of nodes
Node* const head_;
+// When value is true, means indicates that duplicate values are allowed.
Review comment:
This comment is misplace.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2359: Improve SkipList memory usage tracking
kangkaisen commented on a change in pull request #2359: Improve SkipList memory
usage tracking
URL: https://github.com/apache/incubator-doris/pull/2359#discussion_r352983630
##
File path: be/src/olap/skiplist.h
##
@@ -100,9 +101,11 @@ class SkipList {
// Immutable after construction
Comparator const compare_;
+bool _can_dup;
Review comment:
_is_unique is better?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2359: Improve SkipList memory usage tracking
kangkaisen commented on a change in pull request #2359: Improve SkipList memory
usage tracking
URL: https://github.com/apache/incubator-doris/pull/2359#discussion_r352987194
##
File path: be/src/olap/memtable.h
##
@@ -56,7 +61,10 @@ class MemTable {
RowCursorComparator _row_comparator;
std::unique_ptr _mem_tracker;
-std::unique_ptr _mem_pool;
+// This is a buffer, to hold the memory referenced by the rows that have
not
+// been inserted into the SkipList
+std::unique_ptr _tmp_mem_pool;
Review comment:
Would better to explain we add a `_tmp_mem_pool` is in order to make
`MemTable::memory_usage` compute accurately.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg opened a new pull request #2362: [FIX] fix arithmetic operation between numeric and non-numeric
yangzhg opened a new pull request #2362: [FIX] fix arithmetic operation between numeric and non-numeric URL: https://github.com/apache/incubator-doris/pull/2362 fix arithmetic operation between numeric and non-numeric will cause unexpected value (#2142 #2145) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new pull request #2361: Fixed a bug that HttpServer in unit test does not start correctly.
morningman opened a new pull request #2361: Fixed a bug that HttpServer in unit test does not start correctly. URL: https://github.com/apache/incubator-doris/pull/2361 Because the http client in unit test try to connect to the server when server is not ready yet. Add a boolean flag `isStarted` to indicate that the server thread is ready or not. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on issue #2341: Optimization: Reduce the number of segments generated in one import and ensure they do not overlap
kangkaisen commented on issue #2341: Optimization: Reduce the number of segments generated in one import and ensure they do not overlap URL: https://github.com/apache/incubator-doris/issues/2341#issuecomment-560994176 Nice job！ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 opened a new pull request #2360: Support subquery with non-scalar result in Binary predicate and Between-and predicate
EmmyMiao87 opened a new pull request #2360: Support subquery with non-scalar result in Binary predicate and Between-and predicate URL: https://github.com/apache/incubator-doris/pull/2360 This commit add a new plan node named AssertNumRowsNode which is used to determine whether the number of rows exceeds the limit. The subquery in Binary predicate and Between-and predicate should be added a AssertNumRowsNode which is used to determine whether the number of rows in subquery is more than 1. If the number of rows in subquery is more than 1, the query will be cancelled. For example: There are 4 rows in table t1. Query: select c1 from t1 where c1=(select c2 from t1); Result: ERROR 1064 (HY000): Expected no more than 1 to be returned by expression select c2 from t1 ISSUE-2270 TPC-DS 6,54,58 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2237: Optimize Doris On Elasticsearch performance
imay commented on a change in pull request #2237: Optimize Doris On
Elasticsearch performance
URL: https://github.com/apache/incubator-doris/pull/2237#discussion_r352979632
##
File path: be/src/exec/es/es_scroll_query.cpp
##
@@ -75,15 +74,39 @@ std::string ESScrollQueryBuilder::build(const
std::map
BooleanQueryBuilder::to_query(predicates, _document, _node);
// note: add `query` for this value
es_query_dsl.AddMember("query", query_node, allocator);
-// just filter the selected fields for reducing the network cost
-if (fields.size() > 0) {
-rapidjson::Value source_node(rapidjson::kArrayType);
+bool pure_docvalue = true;
+// check docvalue sacan optimization
+if (docvalue_context.size() == 0 || docvalue_context.size() <
fields.size()) {
+pure_docvalue = false;
+} else {
+for (auto iter = fields.begin(); iter != fields.end(); iter++) {
Review comment:
```suggestion
for (auto& field : fields) {
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2237: Optimize Doris On Elasticsearch performance
imay commented on a change in pull request #2237: Optimize Doris On
Elasticsearch performance
URL: https://github.com/apache/incubator-doris/pull/2237#discussion_r352979815
##
File path: be/src/exec/es/es_scroll_query.cpp
##
@@ -75,15 +74,39 @@ std::string ESScrollQueryBuilder::build(const
std::map
BooleanQueryBuilder::to_query(predicates, _document, _node);
// note: add `query` for this value
es_query_dsl.AddMember("query", query_node, allocator);
-// just filter the selected fields for reducing the network cost
-if (fields.size() > 0) {
-rapidjson::Value source_node(rapidjson::kArrayType);
+bool pure_docvalue = true;
+// check docvalue sacan optimization
+if (docvalue_context.size() == 0 || docvalue_context.size() <
fields.size()) {
+pure_docvalue = false;
+} else {
+for (auto iter = fields.begin(); iter != fields.end(); iter++) {
+if (!(docvalue_context.find(*iter) != docvalue_context.end())) {
+pure_docvalue = false;
+break;
+}
+}
+}
+rapidjson::Value source_node(rapidjson::kArrayType);
+if (pure_docvalue) {
+for (auto iter = fields.begin(); iter != fields.end(); iter++) {
Review comment:
```suggestion
for (auto& field : fields) {
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2237: Optimize Doris On Elasticsearch performance
imay commented on a change in pull request #2237: Optimize Doris On
Elasticsearch performance
URL: https://github.com/apache/incubator-doris/pull/2237#discussion_r352979530
##
File path: be/src/exec/es/es_scroll_query.cpp
##
@@ -75,15 +74,39 @@ std::string ESScrollQueryBuilder::build(const
std::map
BooleanQueryBuilder::to_query(predicates, _document, _node);
// note: add `query` for this value
es_query_dsl.AddMember("query", query_node, allocator);
-// just filter the selected fields for reducing the network cost
-if (fields.size() > 0) {
-rapidjson::Value source_node(rapidjson::kArrayType);
+bool pure_docvalue = true;
+// check docvalue sacan optimization
+if (docvalue_context.size() == 0 || docvalue_context.size() <
fields.size()) {
+pure_docvalue = false;
+} else {
+for (auto iter = fields.begin(); iter != fields.end(); iter++) {
+if (!(docvalue_context.find(*iter) != docvalue_context.end())) {
Review comment:
```suggestion
if (docvalue_context.find(*iter) == docvalue_context.end()) {
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] chaoyli edited a comment on issue #2330: Remove VersionHash in BE and FE
chaoyli edited a comment on issue #2330: Remove VersionHash in BE and FE URL: https://github.com/apache/incubator-doris/issues/2330#issuecomment-560349432 Divide the logic into tow procedures. 1. Removing comparison in BE and FE related to version hash. #2335 #2358 2. Modify API in FE/BE. Removing all version hash. Procedure 1 is completed in Doris-0.13, procedure 2 is completed in Doris-0.14. Why removing version hash involve two Doris release? The normal upgrade procedure for Doris is upgrading BE before upgrading FE. If version hash in BE have been removed, upgrading BE will lose version hash info which is useful for FE. So we should removing version hash related to comparison in FE and BE in Doris-0.13 firstly. And Doris-0.13 upgraded, all comparison related to version hash is removed, we can remove version hash in all API in BE and FE. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] chaoyli edited a comment on issue #2330: Remove VersionHash in BE and FE
chaoyli edited a comment on issue #2330: Remove VersionHash in BE and FE URL: https://github.com/apache/incubator-doris/issues/2330#issuecomment-560349432 Divide the logic into tow procedures. 1. Removing comparison in BE and FE related to version hash. #2335 #2358 2. Modify API in FE/BE. Removing all version hash. Procedure 1 is completed in Doris-0.13, procedure 2 is completed in Doris-0.14. Why removing version hash involve two Doris release? The normal upgrade procedure for Doris is upgrading BE before upgrading FE. If version hash in BE have been removed, upgrading BE will lose version hash info which is useful for FE. So we should removing version hash related to comparison in FE and BE in Doris-0.13 firstly. And Doris-0.13 upgraded, all comparison related to version hash is removed, we can remove version hash in all API in BE and FE. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] chaoyli edited a comment on issue #2330: Remove VersionHash in BE and FE
chaoyli edited a comment on issue #2330: Remove VersionHash in BE and FE URL: https://github.com/apache/incubator-doris/issues/2330#issuecomment-560349432 Divide the logic into three procedures. 1. Removing comparison in BE and FE related to version hash. #2335 #2358 2. Modify API in FE/BE. Removing all version hash. Procedure 1 is completed in Doris-0.13, procedure 2 is completed in Doris-0.14. Why removing version hash involve two Doris release? The normal upgrade procedure for Doris is upgrading BE before upgrading FE. If version hash in BE have been removed, upgrading BE will lose version hash info which is useful for FE. So we should removing version hash related to comparison in FE and BE in Doris-0.13 firstly. And Doris-0.13 upgraded, all comparison related to version hash is removed, we can remove version hash in all API in BE and FE. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] chaoyli edited a comment on issue #2330: Remove VersionHash in BE and FE
chaoyli edited a comment on issue #2330: Remove VersionHash in BE and FE URL: https://github.com/apache/incubator-doris/issues/2330#issuecomment-560349432 Divide the logic into three procedures. 1. Removing comparison in BE and FE related to version hash. #2335 #2358 2. Modify API in FE. Removing all version hash. 3. Removing version hash in BE. Procedure 1, 2 is completed in Doris-0.13, procedure 3 is completed in Doris-0.14. Why removing version hash involve two Doris release? The normal upgrade procedure for Doris is upgrading BE before upgrading FE. If version hash in BE have been removed, upgrading BE will lose version hash info which is useful for FE. So we should removing version hash related to comparison in FE and BE in Doris-0.13 firstly. And Doris-0.13 upgraded, all comparison related to version hash is removed, we can remove version hash in all API in BE and FE. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] chaoyli edited a comment on issue #2330: Remove VersionHash in BE and FE
chaoyli edited a comment on issue #2330: Remove VersionHash in BE and FE URL: https://github.com/apache/incubator-doris/issues/2330#issuecomment-560349432 Divide the logic into three procedures. 1. Removing comparison in BE and FE related to version hash. #2335 #2358 2. Modify API in FE. Removing all version hash. 3. Removing version hash in BE. Procedure 1, 2 is completed in Doris-0.13, procedure 3 is completed in Doris-0.14. Why removing version hash involve two Doris release? The normal upgrade procedure for Doris is upgrading BE before upgrading FE. If version hash in BE have been removed, upgrading BE will lose version hash info which is useful for FE. So we should removing version hash in FE firstly. And after all FE have been upgraded, we can remove version hash in BE and upgrade BE process. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] chaoyli edited a comment on issue #2330: Remove VersionHash in BE and FE
chaoyli edited a comment on issue #2330: Remove VersionHash in BE and FE URL: https://github.com/apache/incubator-doris/issues/2330#issuecomment-560349432 Divide the logic into three procedures. 1. Removing comparison in BE and FE related to version hash. #2335 #2358 2. Modify API in FE. Removing all version hash. 3. Removing version hash in BE. Procedure 1, 2 is completed in Doris-0.13, procedure 3 is completed in Doris-0.14. Why removing version hash involve two Doris release? The normal upgrade procedure for Doris is upgrading BE before upgrading FE. If version hash in BE have been removed, upgrading BE will lose version hash info which is useful for FF. So we should removing version hash in FE firstly. And after all FE have been upgraded, we can remove version hash in BE and upgrade BE process. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] lingbin opened a new pull request #2359: Improve SkipList memory usage tracking
lingbin opened a new pull request #2359: Improve SkipList memory usage tracking URL: https://github.com/apache/incubator-doris/pull/2359 #2341 The problem with the current implementation is that all data to be inserted will be counted in memory, but for the aggregation model or some other special cases, not all data will be inserted into `MemTable`, and these data should not be counted in memory. At present, because of the data row(`Tuple`) generated by the upper layer is different from the data row(`Row`) internally represented by the engine, when inserting `MemTable`, the data row must be copied. And, at present, the aggregation function only supports `MemPool` when copying, so even if the data will not be inserted into` SkipList`, `MemPool` is still used (in the future, it can be replaced with an ordinary` Buffer`). However, we reuse the allocated memory in MemPool, that is, we do not reallocate new memory every time. Note: Due to the characteristics of `MemPool` (once inserted, it cannot be partially cleared), the following scenarios may still cause multiple flushes. For example, the aggregation model of a string column is `MAX`, and the data inserted at the same time is in ascending order, then for each data row, it must apply for memory from `MemPool` in `SkipList`, that is, although the old rows in SkipList` will be discarded, the memory occupied will still be counted. I did a test on my development machine: a table with only one tablet and all columns are keys, the original data was 1.1G (9318799 rows), and there were 377745 rows after removing duplicates. It can be found that both the number of files and the query efficiency are greatly improved. before: ``` $ ll storage/data/0/10019/1075020655/ total 4540 -rw--- 1 dev dev 393152 Dec 2 18:42 0204f5404b740288294b21e52b0786adf3be_0_0.dat -rw--- 1 dev dev 1135 Dec 2 18:42 0204f5404b740288294b21e52b0786adf3be_0_0.idx -rw--- 1 dev dev 421660 Dec 2 18:43 0204f5404b740288294b21e52b0786adf3be_10_0.dat -rw--- 1 dev dev 1185 Dec 2 18:43 0204f5404b740288294b21e52b0786adf3be_10_0.idx -rw--- 1 dev dev 184214 Dec 2 18:42 0204f5404b740288294b21e52b0786adf3be_1_0.dat -rw--- 1 dev dev610 Dec 2 18:42 0204f5404b740288294b21e52b0786adf3be_1_0.idx -rw--- 1 dev dev 329181 Dec 2 18:43 0204f5404b740288294b21e52b0786adf3be_11_0.dat -rw--- 1 dev dev935 Dec 2 18:43 0204f5404b740288294b21e52b0786adf3be_11_0.idx -rw--- 1 dev dev 343813 Dec 2 18:43 0204f5404b740288294b21e52b0786adf3be_12_0.dat -rw--- 1 dev dev985 Dec 2 18:43 0204f5404b740288294b21e52b0786adf3be_12_0.idx -rw--- 1 dev dev 315364 Dec 2 18:42 0204f5404b740288294b21e52b0786adf3be_2_0.dat -rw--- 1 dev dev885 Dec 2 18:42 0204f5404b740288294b21e52b0786adf3be_2_0.idx -rw--- 1 dev dev 423806 Dec 2 18:42 0204f5404b740288294b21e52b0786adf3be_3_0.dat -rw--- 1 dev dev 1185 Dec 2 18:42 0204f5404b740288294b21e52b0786adf3be_3_0.idx -rw--- 1 dev dev 294811 Dec 2 18:42 0204f5404b740288294b21e52b0786adf3be_4_0.dat -rw--- 1 dev dev835 Dec 2 18:42 0204f5404b740288294b21e52b0786adf3be_4_0.idx -rw--- 1 dev dev 403241 Dec 2 18:42 0204f5404b740288294b21e52b0786adf3be_5_0.dat -rw--- 1 dev dev 1135 Dec 2 18:42 0204f5404b740288294b21e52b0786adf3be_5_0.idx -rw--- 1 dev dev 350753 Dec 2 18:42 0204f5404b740288294b21e52b0786adf3be_6_0.dat -rw--- 1 dev dev860 Dec 2 18:42 0204f5404b740288294b21e52b0786adf3be_6_0.idx -rw--- 1 dev dev 266966 Dec 2 18:42 0204f5404b740288294b21e52b0786adf3be_7_0.dat -rw--- 1 dev dev735 Dec 2 18:42 0204f5404b740288294b21e52b0786adf3be_7_0.idx -rw--- 1 dev dev 451191 Dec 2 18:42 0204f5404b740288294b21e52b0786adf3be_8_0.dat -rw--- 1 dev dev 1235 Dec 2 18:42 0204f5404b740288294b21e52b0786adf3be_8_0.idx -rw--- 1 dev dev 398439 Dec 2 18:42 0204f5404b740288294b21e52b0786adf3be_9_0.dat -rw--- 1 dev dev 1110 Dec 2 18:42 0204f5404b740288294b21e52b0786adf3be_9_0.idx mysql> select count(*) from xxx_before; +--+ | count(*) | +--+ | 377745 | +--+ 1 row in set (0.91 sec) ``` after:$ ``` $ ll storage/data/0/10013/1075020655/ total 3612 -rw--- 1 dev dev 3328992 Dec 2 18:26 0203d44e5cc72626f95a0b196b52a05c0f8a_0_0.dat -rw--- 1 dev dev8460 Dec 2 18:26
[GitHub] [incubator-doris] chaoyli edited a comment on issue #2330: Remove VersionHash in BE and FE
chaoyli edited a comment on issue #2330: Remove VersionHash in BE and FE URL: https://github.com/apache/incubator-doris/issues/2330#issuecomment-560349432 Divide the logic into three procedures. 1. Removing comparison in BE and FE related to version hash. #2335 #2338 2. Modify API in FE. Removing all version hash. 3. Removing version hash in BE. Procedure 1, 2 is completed in Doris-0.13, procedure 3 is completed in Doris-0.14. Why removing version hash involve two Doris release? The normal upgrade procedure for Doris is upgrading BE before upgrading FE. If version hash in BE have been removed, upgrading BE will lose version hash info which is useful for FF. So we should removing version hash in FE firstly. And after all FE have been upgraded, we can remove version hash in BE and upgrade BE process. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] chaoyli opened a new pull request #2358: Remove VersionHash used to comparison in BE
chaoyli opened a new pull request #2358: Remove VersionHash used to comparison in BE URL: https://github.com/apache/incubator-doris/pull/2358 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352969745
##
File path: be/src/olap/rowset/segment_v2/index_page.h
##
@@ -78,64 +78,80 @@ class IndexPageBuilder {
// is the builder currently between finish() and reset()?
bool _finished = false;
faststring _buffer;
-std::vector _entry_offsets;
+uint32_t _count = 0;
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352969726
##
File path: be/src/olap/rowset/segment_v2/column_reader.cpp
##
@@ -536,17 +518,17 @@ Status DefaultValueColumnIterator::init(const
ColumnIteratorOptions& opts) {
return Status::OK();
}
-Status DefaultValueColumnIterator::next_batch(size_t* n, ColumnBlock* dst) {
+Status DefaultValueColumnIterator::next_batch(size_t* n, ColumnBlockView* dst)
{
+if (dst->is_nullable()) {
+dst->set_null_bits(*n, _is_default_value_null);
+}
+
if (_is_default_value_null) {
-for (int i = 0; i < *n; ++i) {
-dst->set_is_null(i, true);
-}
+dst->advance(*n);
} else {
for (int i = 0; i < *n; ++i) {
-memcpy(dst->mutable_cell_ptr(i), _mem_value.data(), _value_size);
-if (dst->is_nullable()) {
-dst->set_is_null(i, false);
-}
+memcpy(dst->data(), _mem_value.data(), _value_size);
+dst->advance(1);
Review comment:
it's in the for loop, we memcpy and advance one value every time
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352969581
##
File path: be/src/olap/rowset/segment_v2/column_reader.cpp
##
@@ -278,6 +250,17 @@ Status ColumnReader::_load_zone_map_index() {
return Status::OK();
}
+Status ColumnReader::_load_bitmap_index() {
+if (_meta.has_bitmap_index()) {
+const BitmapIndexColumnPB bitmap_index_meta = _meta.bitmap_index();
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352969604
##
File path: be/src/olap/rowset/segment_v2/column_reader.cpp
##
@@ -278,6 +250,17 @@ Status ColumnReader::_load_zone_map_index() {
return Status::OK();
}
+Status ColumnReader::_load_bitmap_index() {
+if (_meta.has_bitmap_index()) {
+const BitmapIndexColumnPB bitmap_index_meta = _meta.bitmap_index();
+_bitmap_index_reader.reset(new
BitmapIndexReader(_file,bitmap_index_meta));
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352969547
##
File path: be/src/olap/rowset/segment_v2/bitmap_index_reader.cpp
##
@@ -17,10 +17,61 @@
#include "olap/rowset/segment_v2/bitmap_index_reader.h"
+#include "olap/types.h"
+
namespace doris {
namespace segment_v2 {
+Status BitmapIndexReader::load() {
+const IndexedColumnMetaPB& dict_meta = _bitmap_index_meta.dict_column();
+const IndexedColumnMetaPB& bitmap_meta =
_bitmap_index_meta.bitmap_column();
+_has_null = _bitmap_index_meta.has_null();
+
+_dict_column_reader.reset(new IndexedColumnReader(_file, dict_meta));
+_bitmap_column_reader.reset(new IndexedColumnReader(_file, bitmap_meta));
+RETURN_IF_ERROR(_dict_column_reader->load());
+RETURN_IF_ERROR(_bitmap_column_reader->load());
+return Status::OK();
+}
+
+Status BitmapIndexReader::new_iterator(BitmapIndexIterator** iterator) {
+*iterator = new BitmapIndexIterator(this);
+return Status::OK();
+}
+
+Status BitmapIndexIterator::seek_dictionary(const void* value, bool*
exact_match) {
+RETURN_IF_ERROR(_dict_column_iter.seek_at_or_after(value, exact_match));
+_current_rowid = _dict_column_iter.get_current_ordinal();
+return Status::OK();
+}
+
+Status BitmapIndexIterator::read_bitmap(rowid_t ordinal, Roaring* result) {
+DCHECK(0 <= ordinal && ordinal < _reader->bitmap_nums());
+
+Slice value;
+uint8_t nullmap;
+size_t num_to_read = 1;
+ColumnBlock block(_reader->type_info(), (uint8_t*) , ,
num_to_read, _pool.get());
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352969529
##
File path: be/src/olap/in_list_predicate.cpp
##
@@ -109,6 +109,39 @@ IN_LIST_PRED_EVALUATE(NotInListPredicate, ==)
IN_LIST_PRED_COLUMN_BLOCK_EVALUATE(InListPredicate, !=)
IN_LIST_PRED_COLUMN_BLOCK_EVALUATE(NotInListPredicate, ==)
+#define IN_LIST_PRED_BITMAP_EVALUATE(CLASS, OP) \
+template \
+Status CLASS::evaluate(const Schema& schema, const
vector& iterators, uint32_t num_rows, Roaring* bitmap)
const { \
+BitmapIndexIterator *iterator = iterators[_column_id]; \
+if (iterator == nullptr) { \
+return Status::OK(); \
+} \
+if (iterator->has_null_bitmap()) { \
+Roaring null_bitmap; \
+RETURN_IF_ERROR(iterator->read_null_bitmap(_bitmap)); \
+*bitmap -= null_bitmap; \
+} \
+Roaring roaring; \
+for (auto value:_values) { \
+bool exact_match; \
+Status s = iterator->seek_dictionary(, _match); \
+rowid_t seeked_ordinal = iterator->current_ordinal(); \
+if (!s.is_not_found()) { \
+if (!s.ok()) { return s; } \
+if (exact_match) { \
+Roaring bitmap; \
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352969510
##
File path: be/src/olap/comparison_predicate.cpp
##
@@ -138,6 +140,84 @@ COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(LessEqualPredicate,
<=)
COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(GreaterPredicate, >)
COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(GreaterEqualPredicate, >=)
+#define BITMAP_COMPARE_EqualPredicate(s, exact_match, seeked_ordinal,
iterator, bitmap, roaring) \
+if (!s.is_not_found()) { \
+if (!s.ok()) { return s; } \
+if (exact_match) { \
+RETURN_IF_ERROR(iterator->read_bitmap(seeked_ordinal,
)); \
+} \
+} \
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352969491
##
File path: be/src/olap/comparison_predicate.cpp
##
@@ -138,6 +140,84 @@ COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(LessEqualPredicate,
<=)
COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(GreaterPredicate, >)
COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(GreaterEqualPredicate, >=)
+#define BITMAP_COMPARE_EqualPredicate(s, exact_match, seeked_ordinal,
iterator, bitmap, roaring) \
+if (!s.is_not_found()) { \
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed pull request #2356: Fixed a bug that Load job's state is incorrect when upgrading from 0.10.x to 0.11.x
morningman closed pull request #2356: Fixed a bug that Load job's state is incorrect when upgrading from 0.10.x to 0.11.x URL: https://github.com/apache/incubator-doris/pull/2356 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new pull request #2356: Fixed a bug that Load job's state is incorrect when upgrading from 0.10.x to 0.11.x
morningman opened a new pull request #2356: Fixed a bug that Load job's state is incorrect when upgrading from 0.10.x to 0.11.x URL: https://github.com/apache/incubator-doris/pull/2356 There is bug in Doris version 0.10.x. When a load job in PENDING or LOADING state was replayed from image (not through the edit log), we forgot to add the corresponding callback id in the CallbackFactory. As a result, the subsequent finish txn edit logs cannot properly finish the job during the replay process. This results in that when the FE restarts, these load jobs that should have been completed are re-entered into the pending state, resulting in repeated submission load tasks. Those wrong images are unrecoverable, so that we have to reset all load jobs in PENDING or LOADING state when restarting FE, depends on its corresponding txn's status, to avoid submit jobs repeatedly. If corresponding txn exist, set load job' state depends on txn's status. If txn does not exist, may be the txn has been removed due to label expiration. So that we don't know the txn is aborted or visible. So we have to set the job's state as UNKNOWN, which need handle it manually. ISSUE #2354 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 closed pull request #2356: Fixed a bug that Load job's state is incorrect when upgrading from 0.10.x to 0.11.x
EmmyMiao87 closed pull request #2356: Fixed a bug that Load job's state is incorrect when upgrading from 0.10.x to 0.11.x URL: https://github.com/apache/incubator-doris/pull/2356 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new pull request #2356: Fixed a bug that Load job's state is incorrect when upgrading from 0.10.x to 0.11.x
morningman opened a new pull request #2356: Fixed a bug that Load job's state is incorrect when upgrading from 0.10.x to 0.11.x URL: https://github.com/apache/incubator-doris/pull/2356 There is bug in Doris version 0.10.x. When a load job in PENDING or LOADING state was replayed from image (not through the edit log), we forgot to add the corresponding callback id in the CallbackFactory. As a result, the subsequent finish txn edit logs cannot properly finish the job during the replay process. This results in that when the FE restarts, these load jobs that should have been completed are re-entered into the pending state, resulting in repeated submission load tasks. Those wrong images are unrecoverable, so that we have to reset all load jobs in PENDING or LOADING state when restarting FE, depends on its corresponding txn's status, to avoid submit jobs repeatedly. If corresponding txn exist, set load job' state depends on txn's status. If txn does not exist, may be the txn has been removed due to label expiration. So that we don't know the txn is aborted or visible. So we have to set the job's state as UNKNOWN, which need handle it manually. ISSUE #2354 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HangyuanLiu opened a new issue #2357: Fe will notify new FE type transfer and quit unexpected
HangyuanLiu opened a new issue #2357: Fe will notify new FE type transfer and quit unexpected URL: https://github.com/apache/incubator-doris/issues/2357 **Describe the bug** When the fe pressure is large, fe will notify new FE type transfer and quit **To Reproduce** 2019-12-03 01:23:04,052 WARN 53 [Catalog.notifyNewFETypeTransfer():2152] notify new FE type transfer: UNKNOWN 2019-12-03 01:23:04,053 INFO 65 [Catalog$4.runOneCycle():2175] begin to transfer FE type from MASTER to UNKNOWN 2019-12-03 01:23:04,053 ERROR 65 [Catalog$4.runOneCycle():2252] transfer FE type from MASTER to UNKNOWN. exit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #2355: Bug fixs: sql mode
morningman commented on a change in pull request #2355: Bug fixs: sql mode URL: https://github.com/apache/incubator-doris/pull/2355#discussion_r352961919 ## File path: fe/src/main/java/org/apache/doris/qe/VariableVarConverters.java ## @@ -0,0 +1,46 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.qe; + +import com.google.common.collect.Maps; +import org.apache.doris.common.DdlException; + +import java.util.Map; + Review comment: Add comment to this class. What is this class for, and how to add new converter? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #2355: Bug fixs: sql mode
morningman commented on a change in pull request #2355: Bug fixs: sql mode
URL: https://github.com/apache/incubator-doris/pull/2355#discussion_r352960732
##
File path: fe/src/main/java/org/apache/doris/qe/SqlModeHelper.java
##
@@ -139,21 +143,26 @@ public static String decode(Long sqlMode) throws
AnalysisException {
}
// convert string type SQL MODE to long type that session can store
-public static Long encode(String sqlMode) throws AnalysisException {
+public static Long encode(String sqlMode) throws DdlException {
List names =
Splitter.on(',').trimResults().omitEmptyStrings().splitToList(sqlMode);
// empty string parse to 0
long value = 0L;
for (String key : names) {
+if (StringUtils.isNumeric(key)) {
+value |= expand(Long.valueOf(key));
+LOG.warn("Set sql mode with numerical value");
Review comment:
Why here is a warning log?
Here should only has debug level log.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #2317: Support Dynamic Partition
morningman commented on issue #2317: Support Dynamic Partition
URL: https://github.com/apache/incubator-doris/pull/2317#issuecomment-560970173
Have a few questions
1. The TableProperty class needs to be more general.
For example, there may be only one variable in class: `Map properties`
For dynamic partition related properties, uniformly use the following
key-value format:
```
"dynamic_partition.prop1" = "val1",
"dynamic_partition.prop2" = "val2"
```
Just put a map, and then different categories of properties are
distinguished with different prefixes.
The map is mainly to facilitate the subsequent addition of other
categories of properties,
and o need to modify the FeMetaVersion.
After that, you can write a class named, for example,
DynamicPartitionProperty, separately.
This class is built from `properties` map, and no need to be persisted
to meta data.
For example, in the `readFields()` method of TableProperty, you can
write:
```
public void readFields () {
... read the property map first ...
this.dynamicPartitionProperty =
buildDynamicPropertyFromPropertyMap(properties);
this.someOtherProperty =
buildSomeOtherPropertyFromPropertyMap(properties);
...
}
```
In subsequent use, you can directly use the parsed content in
dynamicPartitionProperty
without having to get it from the `properties` map and then parse it.
2. In my opinion, the TableProperty class, is a kind of static class, and
only need to be modified.
when the user changes the relevant property's meta-information (such as
modifying the information about how to create dynamic partition).
In other words, you should only modify the contents of the
TableProperty in the operations of
`create a table`, `stop dynamic partitioning`, `modify dynamic
partition metadata`, or `start dynamic partition`.
The `DynamicPartitionScheduler` does not need to modify the
TableProperty class when it automatically creates partitions.
First, after the partition is automatically created, the new partition
information has been saved and persisted in the table's partition information.
Secondly, infos which `ShowDynamicPartitionStmt` needs to be displayed,
such as `LastSchedulerTime`, does not actually need to be persisted.
The user usually only cares about the success of the last time and the
current state, so this information will be available after the FE restarts and
the DynamicPartitionScheduler is executed again.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] FANLONGFANLONG commented on issue #1684: [Proposal] Bitmap Index File Format for V2 segment
FANLONGFANLONG commented on issue #1684: [Proposal] Bitmap Index File Format for V2 segment URL: https://github.com/apache/incubator-doris/issues/1684#issuecomment-560955221 @gaodayue good proposal! May I know how you deal with the delete rows operation? if remove some rows, what happen to posting list and ordered dictionary ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new pull request #2356: Fixed a bug that Load job's state is incorrect when upgrading from 0.10.x to 0.11.x
morningman opened a new pull request #2356: Fixed a bug that Load job's state is incorrect when upgrading from 0.10.x to 0.11.x URL: https://github.com/apache/incubator-doris/pull/2356 There is bug in Doris version 0.10.x. When a load job in PENDING or LOADING state was replayed from image (not through the edit log), we forgot to add the corresponding callback id in the CallbackFactory. As a result, the subsequent finish txn edit logs cannot properly finish the job during the replay process. This results in that when the FE restarts, these load jobs that should have been completed are re-entered into the pending state, resulting in repeated submission load tasks. Those wrong images are unrecoverable, so that we have to reset all load jobs in PENDING or LOADING state when restarting FE, depends on its corresponding txn's status, to avoid submit jobs repeatedly. If corresponding txn exist, set load job' state depends on txn's status. If txn does not exist, may be the txn has been removed due to label expiration. So that we don't know the txn is aborted or visible. So we have to set the job's state as UNKNOWN, which need handle it manually. ISSUE #2354 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2319: Add Bitmap index reader
imay commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352645052
##
File path: be/src/olap/rowset/segment_v2/bitmap_index_reader.cpp
##
@@ -17,10 +17,61 @@
#include "olap/rowset/segment_v2/bitmap_index_reader.h"
+#include "olap/types.h"
+
namespace doris {
namespace segment_v2 {
+Status BitmapIndexReader::load() {
+const IndexedColumnMetaPB& dict_meta = _bitmap_index_meta.dict_column();
+const IndexedColumnMetaPB& bitmap_meta =
_bitmap_index_meta.bitmap_column();
+_has_null = _bitmap_index_meta.has_null();
+
+_dict_column_reader.reset(new IndexedColumnReader(_file, dict_meta));
+_bitmap_column_reader.reset(new IndexedColumnReader(_file, bitmap_meta));
+RETURN_IF_ERROR(_dict_column_reader->load());
+RETURN_IF_ERROR(_bitmap_column_reader->load());
+return Status::OK();
+}
+
+Status BitmapIndexReader::new_iterator(BitmapIndexIterator** iterator) {
+*iterator = new BitmapIndexIterator(this);
+return Status::OK();
+}
+
+Status BitmapIndexIterator::seek_dictionary(const void* value, bool*
exact_match) {
+RETURN_IF_ERROR(_dict_column_iter.seek_at_or_after(value, exact_match));
+_current_rowid = _dict_column_iter.get_current_ordinal();
+return Status::OK();
+}
+
+Status BitmapIndexIterator::read_bitmap(rowid_t ordinal, Roaring* result) {
+DCHECK(0 <= ordinal && ordinal < _reader->bitmap_nums());
+
+Slice value;
+uint8_t nullmap;
+size_t num_to_read = 1;
+ColumnBlock block(_reader->type_info(), (uint8_t*) , ,
num_to_read, _pool.get());
Review comment:
should call _pool.clear() to avoid too much memory footprint
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2319: Add Bitmap index reader
imay commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352652541
##
File path: be/src/olap/rowset/segment_v2/column_reader.cpp
##
@@ -278,6 +250,17 @@ Status ColumnReader::_load_zone_map_index() {
return Status::OK();
}
+Status ColumnReader::_load_bitmap_index() {
+if (_meta.has_bitmap_index()) {
+const BitmapIndexColumnPB bitmap_index_meta = _meta.bitmap_index();
Review comment:
```suggestion
const BitmapIndexColumnPB& bitmap_index_meta = _meta.bitmap_index();
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2319: Add Bitmap index reader
imay commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352637982
##
File path: be/src/olap/comparison_predicate.cpp
##
@@ -138,6 +140,84 @@ COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(LessEqualPredicate,
<=)
COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(GreaterPredicate, >)
COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(GreaterEqualPredicate, >=)
+#define BITMAP_COMPARE_EqualPredicate(s, exact_match, seeked_ordinal,
iterator, bitmap, roaring) \
+if (!s.is_not_found()) { \
+if (!s.ok()) { return s; } \
+if (exact_match) { \
+RETURN_IF_ERROR(iterator->read_bitmap(seeked_ordinal,
)); \
+} \
+} \
Review comment:
better to remove last \
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2319: Add Bitmap index reader
imay commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352639264
##
File path: be/src/olap/in_list_predicate.cpp
##
@@ -109,6 +109,39 @@ IN_LIST_PRED_EVALUATE(NotInListPredicate, ==)
IN_LIST_PRED_COLUMN_BLOCK_EVALUATE(InListPredicate, !=)
IN_LIST_PRED_COLUMN_BLOCK_EVALUATE(NotInListPredicate, ==)
+#define IN_LIST_PRED_BITMAP_EVALUATE(CLASS, OP) \
+template \
+Status CLASS::evaluate(const Schema& schema, const
vector& iterators, uint32_t num_rows, Roaring* bitmap)
const { \
+BitmapIndexIterator *iterator = iterators[_column_id]; \
+if (iterator == nullptr) { \
+return Status::OK(); \
+} \
+if (iterator->has_null_bitmap()) { \
+Roaring null_bitmap; \
+RETURN_IF_ERROR(iterator->read_null_bitmap(_bitmap)); \
+*bitmap -= null_bitmap; \
+} \
+Roaring roaring; \
+for (auto value:_values) { \
+bool exact_match; \
+Status s = iterator->seek_dictionary(, _match); \
+rowid_t seeked_ordinal = iterator->current_ordinal(); \
+if (!s.is_not_found()) { \
+if (!s.ok()) { return s; } \
+if (exact_match) { \
+Roaring bitmap; \
Review comment:
rename it, it was duplicate with input param.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2319: Add Bitmap index reader
imay commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352654301
##
File path: be/src/olap/rowset/segment_v2/column_reader.cpp
##
@@ -536,17 +518,17 @@ Status DefaultValueColumnIterator::init(const
ColumnIteratorOptions& opts) {
return Status::OK();
}
-Status DefaultValueColumnIterator::next_batch(size_t* n, ColumnBlock* dst) {
+Status DefaultValueColumnIterator::next_batch(size_t* n, ColumnBlockView* dst)
{
+if (dst->is_nullable()) {
+dst->set_null_bits(*n, _is_default_value_null);
+}
+
if (_is_default_value_null) {
-for (int i = 0; i < *n; ++i) {
-dst->set_is_null(i, true);
-}
+dst->advance(*n);
} else {
for (int i = 0; i < *n; ++i) {
-memcpy(dst->mutable_cell_ptr(i), _mem_value.data(), _value_size);
-if (dst->is_nullable()) {
-dst->set_is_null(i, false);
-}
+memcpy(dst->data(), _mem_value.data(), _value_size);
+dst->advance(1);
Review comment:
why 1?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2319: Add Bitmap index reader
imay commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352658221
##
File path: be/src/olap/rowset/segment_v2/index_page.h
##
@@ -78,64 +78,80 @@ class IndexPageBuilder {
// is the builder currently between finish() and reset()?
bool _finished = false;
faststring _buffer;
-std::vector _entry_offsets;
+uint32_t _count = 0;
Review comment:
should update comment to keep consistent.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2319: Add Bitmap index reader
imay commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352652744
##
File path: be/src/olap/rowset/segment_v2/column_reader.cpp
##
@@ -278,6 +250,17 @@ Status ColumnReader::_load_zone_map_index() {
return Status::OK();
}
+Status ColumnReader::_load_bitmap_index() {
+if (_meta.has_bitmap_index()) {
+const BitmapIndexColumnPB bitmap_index_meta = _meta.bitmap_index();
+_bitmap_index_reader.reset(new
BitmapIndexReader(_file,bitmap_index_meta));
Review comment:
```suggestion
_bitmap_index_reader.reset(new BitmapIndexReader(_file,
bitmap_index_meta));
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2319: Add Bitmap index reader
imay commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352637846
##
File path: be/src/olap/comparison_predicate.cpp
##
@@ -138,6 +140,84 @@ COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(LessEqualPredicate,
<=)
COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(GreaterPredicate, >)
COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(GreaterEqualPredicate, >=)
+#define BITMAP_COMPARE_EqualPredicate(s, exact_match, seeked_ordinal,
iterator, bitmap, roaring) \
+if (!s.is_not_found()) { \
Review comment:
better to use do { } while (0)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xy720 opened a new pull request #2355: Bug fixs: sql mode
xy720 opened a new pull request #2355: Bug fixs: sql mode URL: https://github.com/apache/incubator-doris/pull/2355 This commit fixs the bug below, FE throws a unexpected exception when encounter a query like : Set sql_mode = '0,PIPES_AS_CONCAT'. and make some change to sql mode analyze process, now the analyze process is no longer put in SetVar.class, but in VariableMgr.class. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2319: Add Bitmap index reader
imay commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352625665
##
File path: be/src/olap/rowset/segment_v2/bitmap_index_reader.cpp
##
@@ -17,10 +17,65 @@
#include "olap/rowset/segment_v2/bitmap_index_reader.h"
+#include "olap/types.h"
+#include "runtime/mem_tracker.h"
+#include "runtime/mem_pool.h"
+
namespace doris {
namespace segment_v2 {
+Status BitmapIndexReader::load() {
+const IndexedColumnMetaPB dict_meta = _bitmap_index_meta.dict_column();
+const IndexedColumnMetaPB bitmap_meta = _bitmap_index_meta.bitmap_column();
+_has_null = _bitmap_index_meta.has_null();
+
+_dict_column_reader.reset(new IndexedColumnReader(_file, dict_meta));
+_bitmap_column_reader.reset(new IndexedColumnReader(_file, bitmap_meta));
+RETURN_IF_ERROR(_dict_column_reader->load());
+RETURN_IF_ERROR(_bitmap_column_reader->load());
+return Status::OK();
+}
+
+Status BitmapIndexReader::new_iterator(BitmapIndexIterator** iterator) {
+*iterator = new BitmapIndexIterator(this);
+return Status::OK();
+}
+
+Status BitmapIndexIterator::seek_dictionary(const void* value, bool*
exact_match) {
+RETURN_IF_ERROR(_dict_column_iter.seek_at_or_after(value, exact_match));
+_current_rowid = _dict_column_iter.get_current_ordinal();
+return Status::OK();
+}
+
+Status BitmapIndexIterator::read_bitmap(rowid_t ordinal, Roaring* result) {
+DCHECK(0 <= ordinal && ordinal < _reader->bitmap_nums());
+
+Slice value;
+uint8_t nullmap;
+MemTracker mem_tracker;
+MemPool mem_pool(_tracker);
+size_t num_to_read = 1;
+ColumnBlock block(get_type_info(OLAP_FIELD_TYPE_VARCHAR), (uint8_t*)
, , num_to_read, _pool);
+ColumnBlockView column_block_view();
+
+RETURN_IF_ERROR(_bitmap_column_iter.seek_to_ordinal(ordinal));
+size_t num_read = num_to_read;
+RETURN_IF_ERROR(_bitmap_column_iter.next_batch(_read,
_block_view));
+DCHECK(num_to_read == num_read);
+*result = Roaring::read(value.data, false);
+return Status::OK();
+}
+
+Status BitmapIndexIterator::read_union_bitmap(rowid_t from, rowid_t to,
Roaring* result) {
+DCHECK(0 <= from && from <= to && to <= _reader->bitmap_nums());
+for (rowid_t pos = from; pos < to; pos++) {
+Roaring bitmap;
+RETURN_IF_ERROR(read_bitmap(pos, ));
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] lingbin closed issue #2315: Show related tablets sample in EXPLAIN
lingbin closed issue #2315: Show related tablets sample in EXPLAIN URL: https://github.com/apache/incubator-doris/issues/2315 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new issue #2354: Load job's state is incorrect when upgrading from 0.10.x to 0.11.x
morningman opened a new issue #2354: Load job's state is incorrect when upgrading from 0.10.x to 0.11.x URL: https://github.com/apache/incubator-doris/issues/2354 There is bug in Doris version 0.10.x. When a load job in PENDING or LOADING state was replayed from image (not through the edit log), we forgot to add the corresponding callback id in the CallbackFactory. As a result, the subsequent finish txn edit logs cannot properly finish the job during the replay process. This results in that when the FE restarts, these load jobs that should have been completed are re-entered into the pending state, resulting in repeated submission load tasks. Those wrong images are unrecoverable, so that we have to cancel all load jobs in PENDING or LOADING state when restarting FE, to avoid submit jobs repeatedly. This code can be remove when upgrading from 0.11.x to future version. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352557717
##
File path: gensrc/proto/segment_v2.proto
##
@@ -220,3 +220,6 @@ message BitmapIndexColumnPB {
optional BitmapType bitmap_type = 6 [default=ROARING_BITMAP];
}
+message BitmapIndexFileFooterPB {
Review comment:
It's a merge issue, remove it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (d90995c -> 875790e)
This is an automated email from the ASF dual-hosted git repository. lichaoyong pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from d90995c Make node info metrics available on all FE node (#2353) add 875790e Remove VersionHash used to comparation in Fe (#2335) No new revisions were added by this update. Summary of changes: .../java/org/apache/doris/catalog/Replica.java | 18 ++-- .../main/java/org/apache/doris/catalog/Tablet.java | 6 +-- .../apache/doris/catalog/TabletInvertedIndex.java | 3 +- .../org/apache/doris/clone/TabletSchedCtx.java | 49 +- .../java/org/apache/doris/load/DeleteInfo.java | 2 +- .../java/org/apache/doris/master/MasterImpl.java | 8 ++-- .../org/apache/doris/master/ReportHandler.java | 9 ++-- .../java/org/apache/doris/task/AgentTaskQueue.java | 3 +- .../java/org/apache/doris/catalog/ReplicaTest.java | 2 +- 9 files changed, 17 insertions(+), 83 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] chaoyli merged pull request #2335: Remove VersionHash used to comparation in Fe
chaoyli merged pull request #2335: Remove VersionHash used to comparation in Fe URL: https://github.com/apache/incubator-doris/pull/2335 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352557145
##
File path: be/src/olap/rowset/segment_v2/bitmap_index_reader.cpp
##
@@ -17,10 +17,65 @@
#include "olap/rowset/segment_v2/bitmap_index_reader.h"
+#include "olap/types.h"
+#include "runtime/mem_tracker.h"
+#include "runtime/mem_pool.h"
+
namespace doris {
namespace segment_v2 {
+Status BitmapIndexReader::load() {
+const IndexedColumnMetaPB dict_meta = _bitmap_index_meta.dict_column();
+const IndexedColumnMetaPB bitmap_meta = _bitmap_index_meta.bitmap_column();
+_has_null = _bitmap_index_meta.has_null();
+
+_dict_column_reader.reset(new IndexedColumnReader(_file, dict_meta));
+_bitmap_column_reader.reset(new IndexedColumnReader(_file, bitmap_meta));
+RETURN_IF_ERROR(_dict_column_reader->load());
+RETURN_IF_ERROR(_bitmap_column_reader->load());
+return Status::OK();
+}
+
+Status BitmapIndexReader::new_iterator(BitmapIndexIterator** iterator) {
+*iterator = new BitmapIndexIterator(this);
+return Status::OK();
+}
+
+Status BitmapIndexIterator::seek_dictionary(const void* value, bool*
exact_match) {
+RETURN_IF_ERROR(_dict_column_iter.seek_at_or_after(value, exact_match));
+_current_rowid = _dict_column_iter.get_current_ordinal();
+return Status::OK();
+}
+
+Status BitmapIndexIterator::read_bitmap(rowid_t ordinal, Roaring* result) {
+DCHECK(0 <= ordinal && ordinal < _reader->bitmap_nums());
+
+Slice value;
+uint8_t nullmap;
+MemTracker mem_tracker;
+MemPool mem_pool(_tracker);
+size_t num_to_read = 1;
+ColumnBlock block(get_type_info(OLAP_FIELD_TYPE_VARCHAR), (uint8_t*)
, , num_to_read, _pool);
+ColumnBlockView column_block_view();
+
+RETURN_IF_ERROR(_bitmap_column_iter.seek_to_ordinal(ordinal));
+size_t num_read = num_to_read;
+RETURN_IF_ERROR(_bitmap_column_iter.next_batch(_read,
_block_view));
+DCHECK(num_to_read == num_read);
+*result = Roaring::read(value.data, false);
+return Status::OK();
+}
+
+Status BitmapIndexIterator::read_union_bitmap(rowid_t from, rowid_t to,
Roaring* result) {
+DCHECK(0 <= from && from <= to && to <= _reader->bitmap_nums());
+for (rowid_t pos = from; pos < to; pos++) {
+Roaring bitmap;
+RETURN_IF_ERROR(read_bitmap(pos, ));
Review comment:
It's mainly improve for memory usage, one bitmap could be large, if we read
all bitmap once, the memory usage maybe very large.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r35242
##
File path: be/src/olap/rowset/segment_v2/bitmap_index_reader.cpp
##
@@ -17,10 +17,65 @@
#include "olap/rowset/segment_v2/bitmap_index_reader.h"
+#include "olap/types.h"
+#include "runtime/mem_tracker.h"
+#include "runtime/mem_pool.h"
+
namespace doris {
namespace segment_v2 {
+Status BitmapIndexReader::load() {
+const IndexedColumnMetaPB dict_meta = _bitmap_index_meta.dict_column();
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352555668
##
File path: be/src/olap/rowset/segment_v2/bitmap_index_reader.cpp
##
@@ -17,10 +17,65 @@
#include "olap/rowset/segment_v2/bitmap_index_reader.h"
+#include "olap/types.h"
+#include "runtime/mem_tracker.h"
+#include "runtime/mem_pool.h"
+
namespace doris {
namespace segment_v2 {
+Status BitmapIndexReader::load() {
+const IndexedColumnMetaPB dict_meta = _bitmap_index_meta.dict_column();
+const IndexedColumnMetaPB bitmap_meta = _bitmap_index_meta.bitmap_column();
+_has_null = _bitmap_index_meta.has_null();
+
+_dict_column_reader.reset(new IndexedColumnReader(_file, dict_meta));
+_bitmap_column_reader.reset(new IndexedColumnReader(_file, bitmap_meta));
+RETURN_IF_ERROR(_dict_column_reader->load());
+RETURN_IF_ERROR(_bitmap_column_reader->load());
+return Status::OK();
+}
+
+Status BitmapIndexReader::new_iterator(BitmapIndexIterator** iterator) {
+*iterator = new BitmapIndexIterator(this);
+return Status::OK();
+}
+
+Status BitmapIndexIterator::seek_dictionary(const void* value, bool*
exact_match) {
+RETURN_IF_ERROR(_dict_column_iter.seek_at_or_after(value, exact_match));
+_current_rowid = _dict_column_iter.get_current_ordinal();
+return Status::OK();
+}
+
+Status BitmapIndexIterator::read_bitmap(rowid_t ordinal, Roaring* result) {
+DCHECK(0 <= ordinal && ordinal < _reader->bitmap_nums());
+
+Slice value;
+uint8_t nullmap;
+MemTracker mem_tracker;
+MemPool mem_pool(_tracker);
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352555474
##
File path: be/src/olap/rowset/segment_v2/index_page.cpp
##
@@ -67,57 +68,63 @@ Status IndexPageBuilder::get_first_key(Slice* key) const {
///
-IndexPageReader::IndexPageReader() : _parsed(false) {
-}
-
Status IndexPageReader::parse(const Slice& data) {
-return Status(); // FIXME
-}
-
-size_t IndexPageReader::count() const {
-CHECK(_parsed) << "not parsed";
-return _footer.num_entries();
-}
+size_t buffer_len = data.size;
+const uint8_t* buffer = (uint8_t*)data.data;
+size_t footer_size = decode_fixed32_le(buffer + buffer_len - 4);
+std::string footer_buf(data.data + buffer_len - 4 - footer_size,
footer_size);
+_footer.ParseFromString(footer_buf);
+size_t num_entries = _footer.num_entries();
+size_t entry_offset_start = buffer_len - 4 - footer_size - num_entries * 4;
+for(size_t i = 0; i < num_entries; i++) {
+size_t entry_offset = decode_fixed32_le(buffer + entry_offset_start);
+_entry_offsets.push_back(entry_offset);
Review comment:
Has removed the entry_offsets from index_page.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352555625
##
File path: be/src/olap/rowset/segment_v2/bitmap_index_reader.cpp
##
@@ -17,10 +17,65 @@
#include "olap/rowset/segment_v2/bitmap_index_reader.h"
+#include "olap/types.h"
+#include "runtime/mem_tracker.h"
+#include "runtime/mem_pool.h"
+
namespace doris {
namespace segment_v2 {
+Status BitmapIndexReader::load() {
+const IndexedColumnMetaPB dict_meta = _bitmap_index_meta.dict_column();
+const IndexedColumnMetaPB bitmap_meta = _bitmap_index_meta.bitmap_column();
+_has_null = _bitmap_index_meta.has_null();
+
+_dict_column_reader.reset(new IndexedColumnReader(_file, dict_meta));
+_bitmap_column_reader.reset(new IndexedColumnReader(_file, bitmap_meta));
+RETURN_IF_ERROR(_dict_column_reader->load());
+RETURN_IF_ERROR(_bitmap_column_reader->load());
+return Status::OK();
+}
+
+Status BitmapIndexReader::new_iterator(BitmapIndexIterator** iterator) {
+*iterator = new BitmapIndexIterator(this);
+return Status::OK();
+}
+
+Status BitmapIndexIterator::seek_dictionary(const void* value, bool*
exact_match) {
+RETURN_IF_ERROR(_dict_column_iter.seek_at_or_after(value, exact_match));
+_current_rowid = _dict_column_iter.get_current_ordinal();
+return Status::OK();
+}
+
+Status BitmapIndexIterator::read_bitmap(rowid_t ordinal, Roaring* result) {
+DCHECK(0 <= ordinal && ordinal < _reader->bitmap_nums());
+
+Slice value;
+uint8_t nullmap;
+MemTracker mem_tracker;
+MemPool mem_pool(_tracker);
+size_t num_to_read = 1;
+ColumnBlock block(get_type_info(OLAP_FIELD_TYPE_VARCHAR), (uint8_t*)
, , num_to_read, _pool);
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352554975
##
File path: be/src/olap/rowset/segment_v2/indexed_column_reader.cpp
##
@@ -17,55 +17,254 @@
#include "olap/rowset/segment_v2/indexed_column_reader.h"
+#include "env/env.h" // for RandomAccessFile
+#include "gutil/strings/substitute.h" // for Substitute
+#include "olap/key_coder.h"
+#include "olap/rowset/segment_v2/encoding_info.h" // for EncodingInfo
+#include "olap/rowset/segment_v2/index_page.h" // for IndexPageReader
+#include "olap/rowset/segment_v2/options.h" // for PageDecoderOptions
+#include "olap/rowset/segment_v2/page_compression.h"
+#include "olap/rowset/segment_v2/page_decoder.h" // for PagePointer
+#include "util/crc32c.h"
+#include "util/rle_encoding.h" // for RleDecoder
+
namespace doris {
namespace segment_v2 {
-IndexedColumnReader::IndexedColumnReader(RandomAccessFile* file) {
-// TODO
-}
-Status IndexedColumnReader::init() {
-return Status(); // TODO
-}
-Status
IndexedColumnReader::new_iterator(std::unique_ptr* iter)
{
-return Status(); // TODO
-}
-Status IndexedColumnReader::read_page(const PagePointer& pp, PageHandle* ret) {
-return Status(); // TODO
-}
-const IndexedColumnMetaPB& IndexedColumnReader::meta() const {
-static IndexedColumnMetaPB temp;
-return temp; // TODO
-}
-bool IndexedColumnReader::has_ordinal_index() const {
-return false; // TODO
+using strings::Substitute;
+
+Status IndexedColumnReader::load() {
+_type_info = get_type_info((FieldType)_meta.data_type());
+if (_type_info == nullptr) {
+return Status::NotSupported(Substitute("unsupported typeinfo,
type=$0", _meta.data_type()));
+}
+RETURN_IF_ERROR(EncodingInfo::get(_type_info, _meta.encoding(),
&_encoding_info));
+RETURN_IF_ERROR(get_block_compression_codec(_meta.compression(),
&_compress_codec));
+_validx_key_coder = get_key_coder(_type_info->type());
+
+// read and parse ordinal index page when exists
+if (_meta.has_ordinal_index_meta()) {
+if (_meta.ordinal_index_meta().is_root_data_page()) {
+_sole_data_page =
PagePointer(_meta.ordinal_index_meta().root_page());
+} else {
+RETURN_IF_ERROR(read_page(_meta.ordinal_index_meta().root_page(),
&_ordinal_index_page_handle));
+
RETURN_IF_ERROR(_ordinal_index_reader.parse(_ordinal_index_page_handle.data()));
+_has_index_page = true;
+}
+}
+
+// read and parse value index page when exists
+if (_meta.has_value_index_meta()) {
+if (_meta.value_index_meta().is_root_data_page()) {
+_sole_data_page =
PagePointer(_meta.value_index_meta().root_page());
+} else {
+RETURN_IF_ERROR(read_page(_meta.value_index_meta().root_page(),
&_value_index_page_handle));
+
RETURN_IF_ERROR(_value_index_reader.parse(_value_index_page_handle.data()));
+_has_index_page = true;
+}
+}
+_num_values = _meta.num_values();
+return Status::OK();
}
-bool IndexedColumnReader::has_value_index() const {
-return false; // TODO
+
+Status IndexedColumnReader::read_page(const PagePointer& pp, PageHandle*
handle) const {
+auto cache = StoragePageCache::instance();
+PageCacheHandle cache_handle;
+StoragePageCache::CacheKey cache_key(_file->file_name(), pp.offset);
+if (cache->lookup(cache_key, _handle)) {
+// we find page in cache, use it
+*handle = PageHandle(std::move(cache_handle));
+return Status::OK();
+}
+// Now we read this from file.
+size_t page_size = pp.size;
+if (page_size < sizeof(uint32_t)) {
+return Status::Corruption(Substitute("Bad page, page size is too
small, size=$0", page_size));
+}
+
+// Now we use this buffer to store page from storage, if this page is
compressed
+// this buffer will assigned uncompressed page, and origin content will be
freed.
+std::unique_ptr page(new uint8_t[page_size]);
+Slice page_slice(page.get(), page_size);
+RETURN_IF_ERROR(_file->read_at(pp.offset, page_slice));
+
+size_t data_size = page_size - 4;
+if (_verify_checksum) {
+uint32_t expect = decode_fixed32_le((uint8_t*)page_slice.data +
page_slice.size - 4);
+uint32_t actual = crc32c::Value(page_slice.data, page_slice.size - 4);
+if (expect != actual) {
+return Status::Corruption(
+Substitute("Page checksum mismatch, actual=$0 vs expect=$1",
actual, expect));
+}
+}
+
+// remove page's suffix
+page_slice.size = data_size;
+if (_compress_codec != nullptr) {
+PageDecompressor decompressor(page_slice, _compress_codec);
+
+Slice uncompressed_page;
+RETURN_IF_ERROR(decompressor.decompress_to(_page));
+
+// If decompressor create new heap memory for uncompressed data,
+//
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352554409
##
File path: be/src/olap/in_list_predicate.cpp
##
@@ -109,6 +111,44 @@ IN_LIST_PRED_EVALUATE(NotInListPredicate, ==)
IN_LIST_PRED_COLUMN_BLOCK_EVALUATE(InListPredicate, !=)
IN_LIST_PRED_COLUMN_BLOCK_EVALUATE(NotInListPredicate, ==)
+#define IN_LIST_PRED_BITMAP_EVALUATE(CLASS, OP) \
+template \
+Status CLASS::evaluate(const Schema& schema, const
vector& iterators, uint32_t num_rows, Roaring* bitmap)
const { \
+BitmapIndexIterator *iterator = iterators[_column_id]; \
+if (iterator == nullptr) { \
+return Status::OK(); \
+} \
+if (iterator->has_null_bitmap()) { \
+Roaring null_bitmap; \
+RETURN_IF_ERROR(iterator->read_null_bitmap(_bitmap)); \
+*bitmap -= null_bitmap; \
+} \
+std::string op = getop(OP); \
+Roaring roaring; \
+for (auto value:_values) { \
+bool exact_match; \
+Status s = iterator->seek_dictionary(, _match); \
+rowid_t seeked_ordinal = iterator->current_ordinal(); \
+if (!s.is_not_found()) { \
+if (!s.ok()) { return s; } \
+if (exact_match) { \
+Roaring bitmap; \
+RETURN_IF_ERROR(iterator->read_bitmap(seeked_ordinal,
)); \
+roaring |= bitmap; \
+} \
+} \
+} \
+if (op.compare("==") == 0) { \
+*bitmap &= roaring; \
+} else { \
+*bitmap -= roaring; \
+} \
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
kangkaisen commented on a change in pull request #2319: Add Bitmap index reader URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352554234 ## File path: be/src/olap/column_predicate.h ## @@ -18,12 +18,18 @@ #ifndef DORIS_BE_SRC_OLAP_COLUMN_PREDICATE_H #define DORIS_BE_SRC_OLAP_COLUMN_PREDICATE_H +#include + #include "olap/column_block.h" #include "olap/selection_vector.h" +#include "olap/rowset/segment_v2/bitmap_index_reader.h" Review comment: ditto This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352554361
##
File path: be/src/olap/field.h
##
@@ -226,6 +226,10 @@ class Field {
void encode_ascending(const void* value, std::string* buf) const {
_key_coder->encode_ascending(value, _index_size, buf);
}
+
+void full_encode_ascending(const void* value, std::string* buf) const {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352554321
##
File path: be/src/olap/comparison_predicate.cpp
##
@@ -138,6 +142,75 @@ COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(LessEqualPredicate,
<=)
COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(GreaterPredicate, >)
COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(GreaterEqualPredicate, >=)
+#define COMPARISON_PRED_BITMAP_EVALUATE(CLASS, OP) \
+template \
+Status CLASS::evaluate(const Schema& schema, const
vector& iterators, uint32_t num_rows, Roaring* bitmap)
const { \
+BitmapIndexIterator *iterator = iterators[_column_id]; \
+if (iterator == nullptr) { \
+return Status::OK(); \
+} \
+std::string op = getop(OP); \
+rowid_t ordinal_limit = iterator->bitmap_nums(); \
+if (iterator->has_null_bitmap()) { \
+ordinal_limit--; \
+Roaring null_bitmap; \
+RETURN_IF_ERROR(iterator->read_null_bitmap(_bitmap)); \
+*bitmap -= null_bitmap; \
+} \
+Roaring roaring; \
+bool exact_match; \
+Status s = iterator->seek_dictionary(&_value, _match); \
+rowid_t seeked_ordinal = iterator->current_ordinal(); \
+if (op.compare("==") == 0) { \
Review comment:
done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r352553188
##
File path: be/src/olap/rowset/segment_v2/segment_writer.cpp
##
@@ -63,6 +63,9 @@ Status SegmentWriter::init(uint32_t write_mbytes_per_sec) {
opts.need_zone_map = true;
}
// TODO set opts.need_bitmap_index based on table properties
+if (!column.is_key()) {
Review comment:
Currently, it's mainly for UT. which column will have inverted index is
decided by FE. After FE work done, this condition could remove.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] chaoyli edited a comment on issue #2330: Remove VersionHash in BE and FE
chaoyli edited a comment on issue #2330: Remove VersionHash in BE and FE URL: https://github.com/apache/incubator-doris/issues/2330#issuecomment-560349432 Divide the logic into three procedures. 1. Removing comparison in FE related to version hash. #2335 2. Modify API in FE. Removing all version hash. 3. Removing version hash in BE. Procedure 1, 2 is completed in Doris-0.13, procedure 3 is completed in Doris-0.14. Why removing version hash involve two Doris release? The normal upgrade procedure for Doris is upgrading BE before upgrading FE. If version hash in BE have been removed, upgrading BE will lose version hash info which is useful for FF. So we should removing version hash in FE firstly. And after all FE have been upgraded, we can remove version hash in BE and upgrade BE process. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on issue #2330: Remove VersionHash in BE and FE
chaoyli commented on issue #2330: Remove VersionHash in BE and FE URL: https://github.com/apache/incubator-doris/issues/2330#issuecomment-560349432 Divide the logic into three procedures. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] Seaven commented on a change in pull request #2351: Add plugin definition
Seaven commented on a change in pull request #2351: Add plugin definition
URL: https://github.com/apache/incubator-doris/pull/2351#discussion_r352496807
##
File path: be/src/plugin/plugin.h
##
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+namespace doris {
+
+#define DORIS_PLUGIN_VERSION 0x0001
+
+typedef struct st_plugin {
Review comment:
OK, will modify and will support example when the feature finish
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: Make node info metrics available on all FE node (#2353)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new d90995c Make node info metrics available on all FE node (#2353)
d90995c is described below
commit d90995c410d06dfaf6dbc6a13bc63695cf5b99b1
Author: Mingyu Chen
AuthorDate: Mon Dec 2 17:31:32 2019 +0800
Make node info metrics available on all FE node (#2353)
Previously, only Master FE has node info metrics to indicate which node is
alive.
But this info should be available on every FE, so that the monitor system
can get all metrics from any FE.
---
fe/src/main/java/org/apache/doris/metric/MetricRepo.java | 6 ++
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/fe/src/main/java/org/apache/doris/metric/MetricRepo.java
b/fe/src/main/java/org/apache/doris/metric/MetricRepo.java
index 9908a1c..68d1cf8 100644
--- a/fe/src/main/java/org/apache/doris/metric/MetricRepo.java
+++ b/fe/src/main/java/org/apache/doris/metric/MetricRepo.java
@@ -293,10 +293,8 @@ public final class MetricRepo {
visitor.visitHistogram(sb, entry.getKey(), entry.getValue());
}
-// master info
-if (Catalog.getInstance().isMaster()) {
-visitor.getNodeInfo(sb);
-}
+// node info
+visitor.getNodeInfo(sb);
return sb.toString();
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #2353: Make node info metrics available on all FE node
morningman merged pull request #2353: Make node info metrics available on all FE node URL: https://github.com/apache/incubator-doris/pull/2353 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new pull request #2353: Make node info metrics available on all FE node
morningman opened a new pull request #2353: Make node info metrics available on all FE node URL: https://github.com/apache/incubator-doris/pull/2353 Previously, only Master FE has node info metrics to indicate which node is alive. But this info should be available on every FE, so that the monitor system can get all metrics from any FE. ISSUE: #2352 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] chaoyli opened a new pull request #2335: Remove VersionHash used to comparation in Fe
chaoyli opened a new pull request #2335: Remove VersionHash used to comparation in Fe URL: https://github.com/apache/incubator-doris/pull/2335 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay closed pull request #2335: Remove VersionHash used to comparation in Fe
imay closed pull request #2335: Remove VersionHash used to comparation in Fe URL: https://github.com/apache/incubator-doris/pull/2335 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2351: Add plugin definition
imay commented on a change in pull request #2351: Add plugin definition URL: https://github.com/apache/incubator-doris/pull/2351#discussion_r352460488 ## File path: be/src/plugin/plugin_manager.h ## @@ -0,0 +1,39 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "string" +#include "unordered_map" + +#include "common/status.h" +#include "plugin.h" Review comment: use full path for inlcude "plugin/plugin.h" This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
