[GitHub] [incubator-doris] morningman commented on a change in pull request #1823: Add a ALTER operation to change distribution type from RANDOM to HASH

[GitBox]

morningman commented on a change in pull request #1823: Add a ALTER operation 
to change distribution type from RANDOM to HASH
URL: https://github.com/apache/incubator-doris/pull/1823#discussion_r325493356
 
 

 ##
 File path: docs/documentation/cn/sql-reference/sql-statements/Data 
Definition/ALTER TABLE.md
 ##
 @@ -232,7 +232,12 @@
 PROPERTIES ("bloom_filter_columns"="k1,k2,k3");
 
 12. 修改表的Colocate 属性
-ALTER TABLE example_db.my_table set ("colocate_with"="t1");
+
+ALTER TABLE example_db.my_table set ("colocate_with" = "t1");
+
+13. 将表的分桶方式由 Random Distribution 改为 Hash Distribution
 
 Review comment:
   I thought about convert it when saving meta image or loading meta image.
   But it is very easy to forget to modify someplace, especially when replaying 
some edit logs related to distribution.
   
   So I think using a command and handle it like a alter table job is more 
safety.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] morningman commented on a change in pull request #1823: Add a ALTER operation to change distribution type from RANDOM to HASH

[GitBox]

morningman commented on a change in pull request #1823: Add a ALTER operation 
to change distribution type from RANDOM to HASH
URL: https://github.com/apache/incubator-doris/pull/1823#discussion_r325492314
 
 

 ##
 File path: fe/src/main/java/org/apache/doris/catalog/OlapTable.java
 ##
 @@ -1115,4 +1115,20 @@ public int getKeysNum() {
 }
 return keysNum;
 }
+
+public boolean convertRandomDistributionToHashDistribution() {
+boolean hasChanged = false;
+List baseSchema = indexIdToSchema.get(baseIndexId);
+if (defaultDistributionInfo.getType() == DistributionInfoType.RANDOM) {
+defaultDistributionInfo = ((RandomDistributionInfo) 
defaultDistributionInfo).toHashDistributionInfo(baseSchema);
+hasChanged = true;
+}
+
+for (Partition partition : idToPartition.values()) {
+if 
(partition.convertRandomDistributionToHashDistribution(baseSchema)) {
 
 Review comment:
   I don't know, just for safety...


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] imay commented on a change in pull request #1823: Add a ALTER operation to change distribution type from RANDOM to HASH

[GitBox]

imay commented on a change in pull request #1823: Add a ALTER operation to 
change distribution type from RANDOM to HASH
URL: https://github.com/apache/incubator-doris/pull/1823#discussion_r325482252
 
 

 ##
 File path: 
fe/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java
 ##
 @@ -46,31 +39,24 @@ public void analyze(Analyzer analyzer) throws 
AnalysisException {
 throw new AnalysisException("Properties is not set");
 }
 
-if (properties.size() == 1 && 
properties.containsKey(KEY_COLOCATE_WITH)) {
-if (Config.disable_colocate_join) {
-throw new AnalysisException("Colocate table is disabled by 
Admin");
-}
-
-if (!Catalog.getCurrentCatalog().getAuth().checkDbPriv(
-ConnectContext.get(), ConnectContext.get().getDatabase(), 
PrivPredicate.ALTER)) {
-
ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR,
-"ALTER");
-}
-} else if 
(!Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(ConnectContext.get(), 
PrivPredicate.ALTER)) {
-
ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR,
-"ALTER");
+if (properties.size() != 1) {
 
 Review comment:
   remove privilege check?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] imay commented on a change in pull request #1816: v2 segment support string encode(#1766)

[GitBox]

imay commented on a change in pull request #1816: v2 segment support string 
encode(#1766)
URL: https://github.com/apache/incubator-doris/pull/1816#discussion_r325480215
 
 

 ##
 File path: be/src/util/arena.cpp
 ##
 @@ -59,6 +60,7 @@ char* Arena::AllocateAligned(size_t bytes) {
 
 char* Arena::AllocateNewBlock(size_t block_bytes) {
 char* result = new char[block_bytes];
+memset(result, 0, block_bytes);
 
 Review comment:
   Yes, I think it's better to only fix column_zone_map's problem, because this 
is used in many other code in which memset is unnecessary.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] imay opened a new issue #1824: All olap scanner hang at ShardedLRUCache's lock

[GitBox]

imay opened a new issue #1824: All olap scanner hang at ShardedLRUCache's lock
URL: https://github.com/apache/incubator-doris/issues/1824
 
 
   Today, we found all backends have no response.
   When I login and check its stack. we found that all scanner are wait lock in 
ShardedLRUCache. The stack looks like following
   ```
   #0  0x7f31aef69e24 in __lll_lock_wait () from 
/opt/compiler/gcc-4.8.2/lib64/libpthread.so.0
   #1  0x7f31aef656d9 in _L_lock_535 () from 
/opt/compiler/gcc-4.8.2/lib64/libpthread.so.0
   #2  0x7f31aef65500 in pthread_mutex_lock () from 
/opt/compiler/gcc-4.8.2/lib64/libpthread.so.0
   #3  0x02192a52 in pthread_mutex_lock ()
   #4  0x00d58630 in doris::Mutex::lock() ()
   #5  0x00d950e3 in doris::ShardedLRUCache::lookup(doris::CacheKey 
const&) ()
   #6  0x00d87d9c in 
doris::FileHandler::open_with_cache(std::__cxx11::basic_string, std::allocator > const&, int) ()
   #7  0x00da1c76 in doris::SegmentReader::_load_segment_file() ()
   #8  0x00da926a in doris::SegmentReader::init(bool) ()
   #9  0x00d68cca in 
doris::ColumnData::_seek_to_block(doris::RowBlockPosition const&, bool) ()
   #10 0x00d69f42 in doris::ColumnData::_get_block ()
   #11 0x00d6c25b in doris::ColumnData::_seek_to_row(doris::RowCursor 
const&, bool, bool) ()
   #12 0x00d6c69c in 
doris::ColumnData::prepare_block_read(doris::RowCursor const*, bool, 
doris::RowCursor const*, bool, doris::RowBlock**) ()
   #13 0x00d20326 in doris::Reader::_attach_data_to_merge_set(bool, 
bool*) ()
   #14 0x00d23476 in doris::Reader::init(doris::ReaderParams const&) ()
   #15 0x0127d54a in doris::OlapScanner::open() ()
   #16 0x01254f94 in 
doris::OlapScanNode::scanner_thread(doris::OlapScanner*) ()
   #17 0x00dfea96 in doris::PriorityThreadPool::work_thread(int) ()
   #18 0x0171cb5d in thread_proxy ()
   #19 0x7f31aef631c3 in start_thread () from 
/opt/compiler/gcc-4.8.2/lib64/libpthread.so.0
   #20 0x7f31af26012d in clone () from 
/opt/compiler/gcc-4.8.2/lib64/libc.so.6
   ```
   And I found that this lock is held to prune FdCache, the stack looks like 
   ```
   #0  0x7f31af25264d in close () from 
/opt/compiler/gcc-4.8.2/lib64/libc.so.6
   #1  0x00d8a8b8 in 
doris::FileHandler::_delete_cache_file_descriptor(doris::CacheKey const&, 
void*) ()
   #2  0x00d95b2a in doris::LRUCache::_unref(doris::LRUHandle*) ()
   #3  0x00d95d3f in doris::ShardedLRUCache::prune() ()
   #4  0x00cba0f8 in doris::OLAPEngine::start_clean_fd_cache() ()
   #5  0x00cf125f in doris::OLAPEngine::_fd_cache_clean_callback(void*) 
()
   #6  0x00cf130f in 
_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5doris10OLAPEngine16_start_bg_workerEvEUlvE4_E6_M_runEv
 ()
   #7  0x026f72ef in execute_native_thread_routine ()
   #8  0x7f31aef631c3 in start_thread () from 
/opt/compiler/gcc-4.8.2/lib64/libpthread.so.0
   #9  0x7f31af26012d in clone () from 
/opt/compiler/gcc-4.8.2/lib64/libc.so.6
   ```
   
   So we should improve our fd cache to avoid such things.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #1818: Add frame_of_reference page

[GitBox]

kangkaisen commented on a change in pull request #1818: Add frame_of_reference 
page
URL: https://github.com/apache/incubator-doris/pull/1818#discussion_r325464393
 
 

 ##
 File path: be/src/util/frame_of_reference_coding.h
 ##
 @@ -0,0 +1,154 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#ifndef DORIS_FRAME_OF_REFERENCE_CODING_H
+#define DORIS_FRAME_OF_REFERENCE_CODING_H
+
+
+#include 
+#include 
+
+#include "util/bit_stream_utils.h"
+#include "util/bit_stream_utils.inline.h"
+#include "util/faststring.h"
+
+namespace doris {
+
+enum {
+BYTE_SIZE = 8,
+INT32_SIZE = 8 * 4,
+FOR_CODING_FRAME_VALUE_NUM = 128
+};
+
+// The implementation for frame-of-reference coding
+// The detail of frame-of-reference coding, please refer to
+// 
https://lemire.me/blog/2012/02/08/effective-compression-using-frame-of-reference-and-delta-coding/
+// and https://www.elastic.co/cn/blog/frame-of-reference-and-roaring-bitmaps
+
+// The encoded data format is as follows:
+
+// 1. Body:
+// BitPackingFrame * FrameCount
+// 2. Footer:
+// FrameOffset(4) * (FrameCount + 1)
+// LastFrameNumValues(1) (optional)
+// FrameCount(4)
+// NumValues(4)
+
+// The BitPackingFrame format is as follows:
+// MinValue, (Value - MinVale) * FRAME_VALUE_NUM
+
+// The last frame value num maybe less than 128
+template
+class ForEncoder {
+public:
+explicit ForEncoder(faststring *buffer): _bit_writer(buffer) {
+}
+
+void put(T value);
+
+// Flushes any pending values to the underlying buffer.
+// Returns the total number of bytes written
+uint32_t flush();
+
+uint32_t len() {
+return _bit_writer.bytes_written();
+}
+
+// Resets all the state in the encoder.
+void clear() {
+_count = 0;
+_frame_count = 0;
+_buffered_values_num = 0;
+_bit_writer.Clear();
+_frame_offsets.clear();
+}
+
+private:
+void bit_packing_buffered_value();
+
+uint32_t _count = 0;
+uint32_t _frame_count = 0;
+uint8_t _buffered_values_num = 0;
+T _buffered_values[FOR_CODING_FRAME_VALUE_NUM];
+
+BitWriter _bit_writer;
+std::vector _frame_offsets;
+};
+
+template
+class ForDecoder {
+public:
+ForDecoder() = default;
+explicit ForDecoder(const uint8_t* in_buffer, size_t buffer_len)
+:_bit_reader(in_buffer, buffer_len) {
 
 Review comment:
   OK


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #1818: Add frame_of_reference page

[GitBox]

kangkaisen commented on a change in pull request #1818: Add frame_of_reference 
page
URL: https://github.com/apache/incubator-doris/pull/1818#discussion_r325463036
 
 

 ##
 File path: be/src/util/frame_of_reference_coding.h
 ##
 @@ -0,0 +1,154 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#ifndef DORIS_FRAME_OF_REFERENCE_CODING_H
+#define DORIS_FRAME_OF_REFERENCE_CODING_H
+
+
+#include 
+#include 
+
+#include "util/bit_stream_utils.h"
+#include "util/bit_stream_utils.inline.h"
+#include "util/faststring.h"
+
+namespace doris {
+
+enum {
+BYTE_SIZE = 8,
+INT32_SIZE = 8 * 4,
+FOR_CODING_FRAME_VALUE_NUM = 128
+};
+
+// The implementation for frame-of-reference coding
+// The detail of frame-of-reference coding, please refer to
+// 
https://lemire.me/blog/2012/02/08/effective-compression-using-frame-of-reference-and-delta-coding/
+// and https://www.elastic.co/cn/blog/frame-of-reference-and-roaring-bitmaps
+
+// The encoded data format is as follows:
+
+// 1. Body:
+// BitPackingFrame * FrameCount
+// 2. Footer:
+// FrameOffset(4) * (FrameCount + 1)
+// LastFrameNumValues(1) (optional)
+// FrameCount(4)
+// NumValues(4)
+
+// The BitPackingFrame format is as follows:
+// MinValue, (Value - MinVale) * FRAME_VALUE_NUM
+
+// The last frame value num maybe less than 128
+template
+class ForEncoder {
+public:
+explicit ForEncoder(faststring *buffer): _bit_writer(buffer) {
+}
+
+void put(T value);
 
 Review comment:
   OK


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #1818: Add frame_of_reference page

[GitBox]

kangkaisen commented on a change in pull request #1818: Add frame_of_reference 
page
URL: https://github.com/apache/incubator-doris/pull/1818#discussion_r325463036
 
 

 ##
 File path: be/src/util/frame_of_reference_coding.h
 ##
 @@ -0,0 +1,154 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#ifndef DORIS_FRAME_OF_REFERENCE_CODING_H
+#define DORIS_FRAME_OF_REFERENCE_CODING_H
+
+
+#include 
+#include 
+
+#include "util/bit_stream_utils.h"
+#include "util/bit_stream_utils.inline.h"
+#include "util/faststring.h"
+
+namespace doris {
+
+enum {
+BYTE_SIZE = 8,
+INT32_SIZE = 8 * 4,
+FOR_CODING_FRAME_VALUE_NUM = 128
+};
+
+// The implementation for frame-of-reference coding
+// The detail of frame-of-reference coding, please refer to
+// 
https://lemire.me/blog/2012/02/08/effective-compression-using-frame-of-reference-and-delta-coding/
+// and https://www.elastic.co/cn/blog/frame-of-reference-and-roaring-bitmaps
+
+// The encoded data format is as follows:
+
+// 1. Body:
+// BitPackingFrame * FrameCount
+// 2. Footer:
+// FrameOffset(4) * (FrameCount + 1)
+// LastFrameNumValues(1) (optional)
+// FrameCount(4)
+// NumValues(4)
+
+// The BitPackingFrame format is as follows:
+// MinValue, (Value - MinVale) * FRAME_VALUE_NUM
+
+// The last frame value num maybe less than 128
+template
+class ForEncoder {
+public:
+explicit ForEncoder(faststring *buffer): _bit_writer(buffer) {
+}
+
+void put(T value);
 
 Review comment:
   OK


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #1818: Add frame_of_reference page

[GitBox]

kangkaisen commented on a change in pull request #1818: Add frame_of_reference 
page
URL: https://github.com/apache/incubator-doris/pull/1818#discussion_r325463007
 
 

 ##
 File path: be/src/util/frame_of_reference_coding.h
 ##
 @@ -0,0 +1,154 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#ifndef DORIS_FRAME_OF_REFERENCE_CODING_H
+#define DORIS_FRAME_OF_REFERENCE_CODING_H
+
+
+#include 
+#include 
+
+#include "util/bit_stream_utils.h"
+#include "util/bit_stream_utils.inline.h"
+#include "util/faststring.h"
+
+namespace doris {
+
+enum {
+BYTE_SIZE = 8,
 
 Review comment:
   OK


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #1818: Add frame_of_reference page

[GitBox]

kangkaisen commented on a change in pull request #1818: Add frame_of_reference 
page
URL: https://github.com/apache/incubator-doris/pull/1818#discussion_r325462969
 
 

 ##
 File path: be/src/util/frame_of_reference_coding.cpp
 ##
 @@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "frame_of_reference_coding.h"
+
+namespace doris {
+
+static inline uint8_t bits(const uint32_t v) {
+return v == 0 ? 0 : 32 - __builtin_clz(v);
+}
+
+template
+void ForEncoder::put(T value) {
+_buffered_values[_buffered_values_num++] = value;
+if (_buffered_values_num == FOR_CODING_FRAME_VALUE_NUM) {
+bit_packing_buffered_value();
+_buffered_values_num = 0;
+_frame_count++;
+}
+_count++;
+}
+
+template
+void ForEncoder::bit_packing_buffered_value() {
+_frame_offsets.push_back(_bit_writer.bits_written());
+
+T min = _buffered_values[0];
+T max = _buffered_values[0];
+for(uint8_t i = 1; i < _buffered_values_num; ++i) {
+if(_buffered_values[i] < min) {
+min = _buffered_values[i];
+}
+if(_buffered_values[i] > max) {
+max = _buffered_values[i];
+}
+}
+uint8_t bit_width = bits(static_cast(max - min));
+
+_bit_writer.PutValue(min, sizeof(T) * BYTE_SIZE);
+for(uint8_t i = 0; i < _buffered_values_num; ++i) {
+_bit_writer.PutValue(_buffered_values[i] - min, bit_width);
+}
+}
+
+template
+uint32_t ForEncoder::flush() {
+if (_buffered_values_num != 0) {
+bit_packing_buffered_value();
+_frame_count++;
+}
+
+// in order to compute the last frame bit_width when decode frame
+_frame_offsets.push_back(_bit_writer.bits_written());
+
+// align to bytes
+_bit_writer.Flush(true);
+
+// write the footer
+for(auto i: _frame_offsets) {
+_bit_writer.PutValue(i, INT32_SIZE);
+}
+
+// write last frame value number
+if (_buffered_values_num != 0) {
+_bit_writer.PutValue(_buffered_values_num,BYTE_SIZE);
+}
+
+_bit_writer.PutValue(_frame_count, INT32_SIZE);
+_bit_writer.PutValue(_count, INT32_SIZE);
+
+_bit_writer.Flush();
+return _bit_writer.bytes_written();
+}
+
+template
+void ForDecoder::decode_current_frame() {
+_current_decoded_frame = _current_index / FOR_CODING_FRAME_VALUE_NUM;
+uint8_t frame_value_num = FOR_CODING_FRAME_VALUE_NUM;
+if (_current_decoded_frame == _frame_count - 1) {
+frame_value_num = _last_frame_num;
+}
+
+uint32_t base_offset = _frame_offsets[_current_decoded_frame];
+_bit_reader.SeekToBit(base_offset);
+
+T min = 0;
+_bit_reader.GetValue(sizeof(T) * BYTE_SIZE, );
+uint8_t bit_width = (_frame_offsets[_current_decoded_frame + 1] - 
base_offset - sizeof(T) * BYTE_SIZE) / frame_value_num;
+
+_out_buffer.clear();
+for(uint8_t i = 0; i < frame_value_num; i ++) {
+uint32_t value;
 
 Review comment:
   it's should be T.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #1818: Add frame_of_reference page

[GitBox]

kangkaisen commented on a change in pull request #1818: Add frame_of_reference 
page
URL: https://github.com/apache/incubator-doris/pull/1818#discussion_r325462838
 
 

 ##
 File path: be/src/util/frame_of_reference_coding.h
 ##
 @@ -0,0 +1,154 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#ifndef DORIS_FRAME_OF_REFERENCE_CODING_H
+#define DORIS_FRAME_OF_REFERENCE_CODING_H
+
+
+#include 
+#include 
+
+#include "util/bit_stream_utils.h"
+#include "util/bit_stream_utils.inline.h"
+#include "util/faststring.h"
+
+namespace doris {
+
+enum {
+BYTE_SIZE = 8,
+INT32_SIZE = 8 * 4,
+FOR_CODING_FRAME_VALUE_NUM = 128
+};
+
+// The implementation for frame-of-reference coding
+// The detail of frame-of-reference coding, please refer to
+// 
https://lemire.me/blog/2012/02/08/effective-compression-using-frame-of-reference-and-delta-coding/
+// and https://www.elastic.co/cn/blog/frame-of-reference-and-roaring-bitmaps
+
+// The encoded data format is as follows:
+
+// 1. Body:
+// BitPackingFrame * FrameCount
+// 2. Footer:
+// FrameOffset(4) * (FrameCount + 1)
+// LastFrameNumValues(1) (optional)
+// FrameCount(4)
+// NumValues(4)
+
+// The BitPackingFrame format is as follows:
+// MinValue, (Value - MinVale) * FRAME_VALUE_NUM
+
+// The last frame value num maybe less than 128
+template
+class ForEncoder {
+public:
+explicit ForEncoder(faststring *buffer): _bit_writer(buffer) {
+}
+
+void put(T value);
 
 Review comment:
   OK


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #1818: Add frame_of_reference page

[GitBox]

kangkaisen commented on a change in pull request #1818: Add frame_of_reference 
page
URL: https://github.com/apache/incubator-doris/pull/1818#discussion_r325462806
 
 

 ##
 File path: be/src/util/frame_of_reference_coding.h
 ##
 @@ -0,0 +1,154 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#ifndef DORIS_FRAME_OF_REFERENCE_CODING_H
+#define DORIS_FRAME_OF_REFERENCE_CODING_H
+
+
+#include 
+#include 
+
+#include "util/bit_stream_utils.h"
+#include "util/bit_stream_utils.inline.h"
+#include "util/faststring.h"
+
+namespace doris {
+
+enum {
+BYTE_SIZE = 8,
+INT32_SIZE = 8 * 4,
+FOR_CODING_FRAME_VALUE_NUM = 128
+};
+
+// The implementation for frame-of-reference coding
+// The detail of frame-of-reference coding, please refer to
+// 
https://lemire.me/blog/2012/02/08/effective-compression-using-frame-of-reference-and-delta-coding/
+// and https://www.elastic.co/cn/blog/frame-of-reference-and-roaring-bitmaps
+
+// The encoded data format is as follows:
+
+// 1. Body:
+// BitPackingFrame * FrameCount
+// 2. Footer:
+// FrameOffset(4) * (FrameCount + 1)
+// LastFrameNumValues(1) (optional)
+// FrameCount(4)
+// NumValues(4)
+
+// The BitPackingFrame format is as follows:
+// MinValue, (Value - MinVale) * FRAME_VALUE_NUM
+
+// The last frame value num maybe less than 128
+template
+class ForEncoder {
+public:
+explicit ForEncoder(faststring *buffer): _bit_writer(buffer) {
+}
+
+void put(T value);
+
+// Flushes any pending values to the underlying buffer.
+// Returns the total number of bytes written
+uint32_t flush();
+
+uint32_t len() {
+return _bit_writer.bytes_written();
+}
+
+// Resets all the state in the encoder.
+void clear() {
+_count = 0;
+_frame_count = 0;
+_buffered_values_num = 0;
+_bit_writer.Clear();
+_frame_offsets.clear();
+}
+
+private:
+void bit_packing_buffered_value();
+
+uint32_t _count = 0;
+uint32_t _frame_count = 0;
+uint8_t _buffered_values_num = 0;
+T _buffered_values[FOR_CODING_FRAME_VALUE_NUM];
+
+BitWriter _bit_writer;
+std::vector _frame_offsets;
+};
+
+template
+class ForDecoder {
+public:
+ForDecoder() = default;
+explicit ForDecoder(const uint8_t* in_buffer, size_t buffer_len)
+:_bit_reader(in_buffer, buffer_len) {
+_bit_reader.SeekToBit(buffer_len * BYTE_SIZE - 8 * BYTE_SIZE);
+_bit_reader.GetValue(INT32_SIZE, &_frame_count);
+_bit_reader.GetValue(INT32_SIZE, &_row_count);
+
+if (_row_count % FOR_CODING_FRAME_VALUE_NUM != 0) {
+_bit_reader.SeekToBit(buffer_len * BYTE_SIZE - 8 * BYTE_SIZE - 8);
+_bit_reader.GetValue(BYTE_SIZE, &_last_frame_num);
+}
+
+size_t frame_offsets_start = 0;
+if (_row_count % FOR_CODING_FRAME_VALUE_NUM != 0) {
+frame_offsets_start = buffer_len * BYTE_SIZE - (_frame_count + 1) 
* INT32_SIZE - BYTE_SIZE * 8 - BYTE_SIZE;
+} else {
+frame_offsets_start = buffer_len * BYTE_SIZE - (_frame_count + 1) 
* INT32_SIZE - BYTE_SIZE * 8;
+}
+_bit_reader.SeekToBit(frame_offsets_start);
+
+for(int i = 0; i < _frame_count + 1; i++ ) {
+uint32_t offset;
+_bit_reader.GetValue(INT32_SIZE, );
+_frame_offsets.push_back(offset);
+}
+};
+
+// Gets the next value.  Returns false if there are no more.
+bool get(T* val);
 
 Review comment:
   OK


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #1818: Add frame_of_reference page

[GitBox]

kangkaisen commented on a change in pull request #1818: Add frame_of_reference 
page
URL: https://github.com/apache/incubator-doris/pull/1818#discussion_r325462497
 
 

 ##
 File path: be/src/util/frame_of_reference_coding.cpp
 ##
 @@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "frame_of_reference_coding.h"
+
+namespace doris {
+
+static inline uint8_t bits(const uint32_t v) {
+return v == 0 ? 0 : 32 - __builtin_clz(v);
+}
+
+template
+void ForEncoder::put(T value) {
+_buffered_values[_buffered_values_num++] = value;
+if (_buffered_values_num == FOR_CODING_FRAME_VALUE_NUM) {
+bit_packing_buffered_value();
+_buffered_values_num = 0;
+_frame_count++;
+}
+_count++;
+}
+
+template
+void ForEncoder::bit_packing_buffered_value() {
+_frame_offsets.push_back(_bit_writer.bits_written());
+
+T min = _buffered_values[0];
+T max = _buffered_values[0];
+for(uint8_t i = 1; i < _buffered_values_num; ++i) {
+if(_buffered_values[i] < min) {
 
 Review comment:
   OK


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #1818: Add frame_of_reference page

[GitBox]

kangkaisen commented on a change in pull request #1818: Add frame_of_reference 
page
URL: https://github.com/apache/incubator-doris/pull/1818#discussion_r325462529
 
 

 ##
 File path: be/src/util/frame_of_reference_coding.cpp
 ##
 @@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "frame_of_reference_coding.h"
+
+namespace doris {
+
+static inline uint8_t bits(const uint32_t v) {
+return v == 0 ? 0 : 32 - __builtin_clz(v);
+}
+
+template
+void ForEncoder::put(T value) {
+_buffered_values[_buffered_values_num++] = value;
+if (_buffered_values_num == FOR_CODING_FRAME_VALUE_NUM) {
+bit_packing_buffered_value();
+_buffered_values_num = 0;
+_frame_count++;
+}
+_count++;
+}
+
+template
+void ForEncoder::bit_packing_buffered_value() {
+_frame_offsets.push_back(_bit_writer.bits_written());
+
+T min = _buffered_values[0];
+T max = _buffered_values[0];
+for(uint8_t i = 1; i < _buffered_values_num; ++i) {
+if(_buffered_values[i] < min) {
+min = _buffered_values[i];
 
 Review comment:
   OK


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #1818: Add frame_of_reference page

[GitBox]

kangkaisen commented on a change in pull request #1818: Add frame_of_reference 
page
URL: https://github.com/apache/incubator-doris/pull/1818#discussion_r325461776
 
 

 ##
 File path: be/src/util/frame_of_reference_coding.cpp
 ##
 @@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "frame_of_reference_coding.h"
 
 Review comment:
   OK


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #1818: Add frame_of_reference page

[GitBox]

kangkaisen commented on a change in pull request #1818: Add frame_of_reference 
page
URL: https://github.com/apache/incubator-doris/pull/1818#discussion_r325461843
 
 

 ##
 File path: be/src/util/frame_of_reference_coding.cpp
 ##
 @@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "frame_of_reference_coding.h"
+
+namespace doris {
+
+static inline uint8_t bits(const uint32_t v) {
 
 Review comment:
   Yes


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] morningman closed pull request #1798: Optimize the load performance for large file

[GitBox]

morningman closed pull request #1798: Optimize the load performance for large 
file
URL: https://github.com/apache/incubator-doris/pull/1798
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] morningman opened a new pull request #1798: Optimize the load performance for large file

[GitBox]

morningman opened a new pull request #1798: Optimize the load performance for 
large file
URL: https://github.com/apache/incubator-doris/pull/1798
 
 
   ISSUE: #1797 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] imay commented on a change in pull request #1798: Optimize the load performance for large file

[GitBox]

imay commented on a change in pull request #1798: Optimize the load performance 
for large file
URL: https://github.com/apache/incubator-doris/pull/1798#discussion_r325455731
 
 

 ##
 File path: be/src/olap/delta_writer.cpp
 ##
 @@ -132,8 +142,10 @@ OLAPStatus DeltaWriter::init() {
 
 _tablet_schema = &(_tablet->tablet_schema());
 _schema = new Schema(*_tablet_schema);
-_mem_table = new MemTable(_schema, _tablet_schema, _req.slots,
-  _req.tuple_desc, _tablet->keys_type());
+_mem_table = std::make_shared(_tablet->tablet_id(), _schema, 
_tablet_schema, _req.slots, _req.tuple_desc, _tablet->keys_type());
+
+_flush_queue_idx = 
_flush_executor->get_queue_idx(_tablet->data_dir()->path_hash());
 
 Review comment:
   Why are you exposing this queue to the outside?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] wangbo commented on a change in pull request #1816: v2 segment support string encode(#1766)

[GitBox]

wangbo commented on a change in pull request #1816: v2 segment support string 
encode(#1766)
URL: https://github.com/apache/incubator-doris/pull/1816#discussion_r325452758
 
 

 ##
 File path: be/src/olap/rowset/segment_v2/binary_dict_page.cpp
 ##
 @@ -102,7 +102,16 @@ Slice BinaryDictPageBuilder::finish() {
 Slice data_slice = _data_page_builder->finish();
 _buffer.append(data_slice.data, data_slice.size);
 encode_fixed32_le(&_buffer[0], _encoding_type);
-return Slice(_buffer.data(), _buffer.size());
+
+if (_encoding_type == DICT_ENCODING) {
+size_t dict_offset = _buffer.size();
+Slice dictionary_page;
+get_dictionary_page(_page);
 
 Review comment:
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] imay commented on a change in pull request #1816: v2 segment support string encode(#1766)

[GitBox]

imay commented on a change in pull request #1816: v2 segment support string 
encode(#1766)
URL: https://github.com/apache/incubator-doris/pull/1816#discussion_r325262158
 
 

 ##
 File path: be/src/util/arena.cpp
 ##
 @@ -59,6 +60,7 @@ char* Arena::AllocateAligned(size_t bytes) {
 
 char* Arena::AllocateNewBlock(size_t block_bytes) {
 char* result = new char[block_bytes];
+memset(result, 0, block_bytes);
 
 Review comment:
   Why memset this to 0? This will cost CPU, we should avoid unnecessary memset 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] imay commented on a change in pull request #1818: Add frame_of_reference page

[GitBox]

imay commented on a change in pull request #1818: Add frame_of_reference page
URL: https://github.com/apache/incubator-doris/pull/1818#discussion_r325205667
 
 

 ##
 File path: be/src/util/frame_of_reference_coding.cpp
 ##
 @@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "frame_of_reference_coding.h"
+
+namespace doris {
+
+static inline uint8_t bits(const uint32_t v) {
 
 Review comment:
   If v is uint64_t, this should be uint64_t?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] imay commented on a change in pull request #1818: Add frame_of_reference page

[GitBox]

imay commented on a change in pull request #1818: Add frame_of_reference page
URL: https://github.com/apache/incubator-doris/pull/1818#discussion_r325245060
 
 

 ##
 File path: be/src/util/frame_of_reference_coding.h
 ##
 @@ -0,0 +1,154 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#ifndef DORIS_FRAME_OF_REFERENCE_CODING_H
+#define DORIS_FRAME_OF_REFERENCE_CODING_H
+
+
+#include 
+#include 
+
+#include "util/bit_stream_utils.h"
+#include "util/bit_stream_utils.inline.h"
+#include "util/faststring.h"
+
+namespace doris {
+
+enum {
+BYTE_SIZE = 8,
+INT32_SIZE = 8 * 4,
+FOR_CODING_FRAME_VALUE_NUM = 128
+};
+
+// The implementation for frame-of-reference coding
+// The detail of frame-of-reference coding, please refer to
+// 
https://lemire.me/blog/2012/02/08/effective-compression-using-frame-of-reference-and-delta-coding/
+// and https://www.elastic.co/cn/blog/frame-of-reference-and-roaring-bitmaps
+
+// The encoded data format is as follows:
+
+// 1. Body:
+// BitPackingFrame * FrameCount
+// 2. Footer:
+// FrameOffset(4) * (FrameCount + 1)
+// LastFrameNumValues(1) (optional)
+// FrameCount(4)
+// NumValues(4)
+
+// The BitPackingFrame format is as follows:
+// MinValue, (Value - MinVale) * FRAME_VALUE_NUM
+
+// The last frame value num maybe less than 128
+template
+class ForEncoder {
+public:
+explicit ForEncoder(faststring *buffer): _bit_writer(buffer) {
+}
+
+void put(T value);
+
+// Flushes any pending values to the underlying buffer.
+// Returns the total number of bytes written
+uint32_t flush();
+
+uint32_t len() {
+return _bit_writer.bytes_written();
+}
+
+// Resets all the state in the encoder.
+void clear() {
+_count = 0;
+_frame_count = 0;
+_buffered_values_num = 0;
+_bit_writer.Clear();
+_frame_offsets.clear();
+}
+
+private:
+void bit_packing_buffered_value();
+
+uint32_t _count = 0;
+uint32_t _frame_count = 0;
+uint8_t _buffered_values_num = 0;
+T _buffered_values[FOR_CODING_FRAME_VALUE_NUM];
+
+BitWriter _bit_writer;
+std::vector _frame_offsets;
+};
+
+template
+class ForDecoder {
+public:
+ForDecoder() = default;
+explicit ForDecoder(const uint8_t* in_buffer, size_t buffer_len)
+:_bit_reader(in_buffer, buffer_len) {
+_bit_reader.SeekToBit(buffer_len * BYTE_SIZE - 8 * BYTE_SIZE);
+_bit_reader.GetValue(INT32_SIZE, &_frame_count);
+_bit_reader.GetValue(INT32_SIZE, &_row_count);
+
+if (_row_count % FOR_CODING_FRAME_VALUE_NUM != 0) {
+_bit_reader.SeekToBit(buffer_len * BYTE_SIZE - 8 * BYTE_SIZE - 8);
+_bit_reader.GetValue(BYTE_SIZE, &_last_frame_num);
+}
+
+size_t frame_offsets_start = 0;
+if (_row_count % FOR_CODING_FRAME_VALUE_NUM != 0) {
+frame_offsets_start = buffer_len * BYTE_SIZE - (_frame_count + 1) 
* INT32_SIZE - BYTE_SIZE * 8 - BYTE_SIZE;
+} else {
+frame_offsets_start = buffer_len * BYTE_SIZE - (_frame_count + 1) 
* INT32_SIZE - BYTE_SIZE * 8;
+}
+_bit_reader.SeekToBit(frame_offsets_start);
+
+for(int i = 0; i < _frame_count + 1; i++ ) {
+uint32_t offset;
+_bit_reader.GetValue(INT32_SIZE, );
+_frame_offsets.push_back(offset);
+}
+};
+
+// Gets the next value.  Returns false if there are no more.
+bool get(T* val);
 
 Review comment:
   Why not provide a get_batch interface?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] imay commented on a change in pull request #1818: Add frame_of_reference page

[GitBox]

imay commented on a change in pull request #1818: Add frame_of_reference page
URL: https://github.com/apache/incubator-doris/pull/1818#discussion_r325247086
 
 

 ##
 File path: be/src/util/frame_of_reference_coding.h
 ##
 @@ -0,0 +1,154 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#ifndef DORIS_FRAME_OF_REFERENCE_CODING_H
+#define DORIS_FRAME_OF_REFERENCE_CODING_H
+
+
+#include 
+#include 
+
+#include "util/bit_stream_utils.h"
+#include "util/bit_stream_utils.inline.h"
+#include "util/faststring.h"
+
+namespace doris {
+
+enum {
+BYTE_SIZE = 8,
 
 Review comment:
   This may conflict with other definition.
   You can define this like following
   ```
   struct ForEncoding {
   enum {
   };
   };
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] imay commented on a change in pull request #1818: Add frame_of_reference page

[GitBox]

imay commented on a change in pull request #1818: Add frame_of_reference page
URL: https://github.com/apache/incubator-doris/pull/1818#discussion_r325208083
 
 

 ##
 File path: be/src/util/frame_of_reference_coding.cpp
 ##
 @@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "frame_of_reference_coding.h"
+
+namespace doris {
+
+static inline uint8_t bits(const uint32_t v) {
+return v == 0 ? 0 : 32 - __builtin_clz(v);
+}
+
+template
+void ForEncoder::put(T value) {
+_buffered_values[_buffered_values_num++] = value;
+if (_buffered_values_num == FOR_CODING_FRAME_VALUE_NUM) {
+bit_packing_buffered_value();
+_buffered_values_num = 0;
+_frame_count++;
+}
+_count++;
+}
+
+template
+void ForEncoder::bit_packing_buffered_value() {
+_frame_offsets.push_back(_bit_writer.bits_written());
+
+T min = _buffered_values[0];
+T max = _buffered_values[0];
+for(uint8_t i = 1; i < _buffered_values_num; ++i) {
+if(_buffered_values[i] < min) {
 
 Review comment:
   ```suggestion
   if (_buffered_values[i] < min) {
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] imay commented on a change in pull request #1818: Add frame_of_reference page

[GitBox]

imay commented on a change in pull request #1818: Add frame_of_reference page
URL: https://github.com/apache/incubator-doris/pull/1818#discussion_r325179899
 
 

 ##
 File path: be/src/util/frame_of_reference_coding.cpp
 ##
 @@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "frame_of_reference_coding.h"
 
 Review comment:
   ```suggestion
   #include "util/frame_of_reference_coding.h"
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] imay commented on a change in pull request #1821: Add some profile metrics

[GitBox]

imay commented on a change in pull request #1821: Add some profile metrics
URL: https://github.com/apache/incubator-doris/pull/1821#discussion_r325175984
 
 

 ##
 File path: be/src/util/runtime_profile.h
 ##
 @@ -64,6 +67,7 @@ namespace doris {
 #define SCOPED_RAW_TIMER(c)
 #define COUNTER_UPDATE(c, v)
 #define COUNTER_SET(c, v)
+#define SET_CUR_TIME(c)
 
 Review comment:
   should add #define ADD_TS(profile, name)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] morningman commented on a change in pull request #1819: Check file descriptor number should between configuration upon start

[GitBox]

morningman commented on a change in pull request #1819: Check file descriptor 
number should between configuration upon start
URL: https://github.com/apache/incubator-doris/pull/1819#discussion_r325169785
 
 

 ##
 File path: docs/documentation/cn/installing/install-deploy.md
 ##
 @@ -378,3 +378,9 @@ Broker 是无状态的进程，可以随意启停。当然，停止后，正在
> 出现这个问题的主要原因是：当用户通过 `ADD BACKEND` 语句添加 BE 后，FE 会识别该语句中指定的是 hostname 还是 
IP。如果是 hostname，则 FE 会自动将其转换为 IP 地址并存储到元数据中。当 BE 在汇报任务完成信息时，会携带自己的 IP 地址。而如果 FE 
发现 BE 汇报的 IP 地址和元数据中不一致时，就会出现如上错误。  
 >
> 这个错误的解决方法：1）分别在 FE 和 BE 设置 **priority\_network** 参数。通常 FE 和 BE 
都处于一个网段，所以该参数设置为相同即可。2）在 `ADD BACKEND` 语句中直接填写 BE 正确的 IP 地址而不是 hostname，以避免 FE 
获取到错误的 IP 地址。
+
+5. BE 进程文件句柄数
+
+BE进程文件句柄数，受min_file_descriptor_number/max_file_descriptor_number两个参数控制。
 
 Review comment:
   You should state the default value of these 2 configurations here, And how 
to modify the `max open files`(eg. by using `ulimit`).
   Or user will not know what's going on and how to fix it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] morningman commented on a change in pull request #1821: Add some profile metrics

[GitBox]

morningman commented on a change in pull request #1821: Add some profile metrics
URL: https://github.com/apache/incubator-doris/pull/1821#discussion_r325165822
 
 

 ##
 File path: be/src/util/runtime_profile.h
 ##
 @@ -108,6 +112,10 @@ class RuntimeProfile {
 
 virtual void set(int value) { _value.store(value); }
 
+   virtual void set_cur_time() {
 
 Review comment:
   Modify the indent


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] morningman commented on a change in pull request #1821: Add some profile metrics

[GitBox]

morningman commented on a change in pull request #1821: Add some profile metrics
URL: https://github.com/apache/incubator-doris/pull/1821#discussion_r325165492
 
 

 ##
 File path: be/src/util/pretty_printer.h
 ##
 @@ -145,7 +145,23 @@ class PrettyPrinter {
 ss << std::setprecision(PRECISION) << output << " ";
 break;
 }
+case TUnit::DATE_US: {
+std::stringstream time_ss;
+   uint64_t time = (uint64_t)value;
 
 Review comment:
   ```suggestion
   uint64_t time = (uint64_t)value;
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[incubator-doris] branch master updated (c4e28f0 -> 3f63bde)

[zhaoc]

This is an automated email from the ASF dual-hosted git repository.

zhaoc pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git.


from c4e28f0  Update FeConstants meta version to VERSION_62 (#1822)
 add 3f63bde  Fix 'Invalid Column Name' error when loading parquet file 
(#1820)

No new revisions were added by this update.

Summary of changes:
 fe/src/main/java/org/apache/doris/load/Load.java | 1 +
 1 file changed, 1 insertion(+)


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] imay merged pull request #1820: Fix 'Invalid Column Name' error when loading parquet file

[GitBox]

imay merged pull request #1820: Fix 'Invalid Column Name' error when loading 
parquet file
URL: https://github.com/apache/incubator-doris/pull/1820
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] xionglei0 removed a comment on issue #1809: support udtf：user defined table generation function

[GitBox]

xionglei0 removed a comment on issue #1809: support udtf：user defined table 
generation function
URL: 
https://github.com/apache/incubator-doris/issues/1809#issuecomment-532144003
 
 
   https://github.com/apache/incubator-doris/issues/1809


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[incubator-doris] branch master updated (dc813e6 -> c4e28f0)

[zhaoc]

This is an automated email from the ASF dual-hosted git repository.

zhaoc pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git.


from dc813e6  Limit the max version to cumulative compaction (#1813)
 add c4e28f0  Update FeConstants meta version to VERSION_62 (#1822)

No new revisions were added by this update.

Summary of changes:
 fe/src/main/java/org/apache/doris/common/FeConstants.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] imay merged pull request #1822: Update FeConstants meta version to VERSION_62

[GitBox]

imay merged pull request #1822: Update FeConstants meta version to VERSION_62
URL: https://github.com/apache/incubator-doris/pull/1822
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] morningman opened a new pull request #1822: Update FeConstants meta version to VERSION_62

[GitBox]

morningman opened a new pull request #1822: Update FeConstants meta version to 
VERSION_62
URL: https://github.com/apache/incubator-doris/pull/1822
 
 
   This should be modified along with commit a232a56c0


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] xionglei0 opened a new pull request #1821: Add some profile metrics

[GitBox]

xionglei0 opened a new pull request #1821: Add some profile metrics
URL: https://github.com/apache/incubator-doris/pull/1821
 
 
   add three metrics in profile:
   
   1. ExecStartTime: when fragment start execute ,display like this: -MM-dd 
HH:mm:ss.SS
   2. ExecEndTime: when fragment finish execute
   3. ReadFilesCount: how many segment files olap_scan_node read
   
   
![image](https://user-images.githubusercontent.com/55084968/65029290-10077700-d970-11e9-84e5-ca5e19e5855a.png)
   
![image](https://user-images.githubusercontent.com/55084968/65029326-201f5680-d970-11e9-9de4-1f19e7f44b31.png)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] nick-yangg commented on issue #1814: 这种FE扩容状态是正常的吗

[GitBox]

nick-yangg commented on issue #1814: 这种FE扩容状态是正常的吗
URL: 
https://github.com/apache/incubator-doris/issues/1814#issuecomment-532128568
 
 
   非常感谢您的回复，我司已经解决了此问题，并修复了此bug。
   
   Mingyu Chen  于2019年9月17日周二 下午2:26写道：
   
   > It not right, the Join status is false, means those 2 Frontends does not
   > join the cluster.
   > You need to check fe.log for more information.
   >
   > —
   > You are receiving this because you authored the thread.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or mute the thread
   > 

   > .
   >
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] EmmyMiao87 commented on issue #1820: Fix 'Invalid Column Name' error when loading parquet file

[GitBox]

EmmyMiao87 commented on issue #1820: Fix 'Invalid Column Name' error when 
loading parquet file
URL: https://github.com/apache/incubator-doris/pull/1820#issuecomment-532110037
 
 
   LGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] morningman opened a new pull request #1820: Fix 'Invalid Column Name' error when loading parquet file

[GitBox]

morningman opened a new pull request #1820: Fix 'Invalid Column Name' error 
when loading parquet file
URL: https://github.com/apache/incubator-doris/pull/1820
 
 
   SlotDescriptor should be with column name, so that ParquetReader can find 
the column by name.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] DarrenChanChenChi commented on issue #1815: mini load：too many filtered rows

[GitBox]

DarrenChanChenChi commented on issue #1815: mini load：too many filtered rows
URL: 
https://github.com/apache/incubator-doris/issues/1815#issuecomment-532103145
 
 
   > @DarrenChanChenChi
   > 
   > Data has been filtered because of some reason. You can visit the URL 
returned by `SHOW LOAD` command. In this URL, there will be detail error 
information.
   
   Thank you! The reason is some column is null，but table‘s column must be not 
null. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] chaoyli opened a new pull request #1819: Check file descriptor number is larger than 65536 upon start

[GitBox]

chaoyli opened a new pull request #1819: Check file descriptor number is larger 
than 65536 upon start
URL: https://github.com/apache/incubator-doris/pull/1819
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] kangkaisen commented on issue #1393: Schema change job could not finished

[GitBox]

kangkaisen commented on issue #1393: Schema change job could not finished
URL: 
https://github.com/apache/incubator-doris/issues/1393#issuecomment-532087958
 
 
   This PR https://github.com/apache/incubator-doris/pull/1695 should fix this 
issue.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] kangkaisen closed issue #1393: Schema change job could not finished

[GitBox]

kangkaisen closed issue #1393: Schema change job could not finished
URL: https://github.com/apache/incubator-doris/issues/1393
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] kangkaisen closed issue #1125: BE core dump when MemTable::insert

[GitBox]

kangkaisen closed issue #1125: BE core dump when MemTable::insert
URL: https://github.com/apache/incubator-doris/issues/1125
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] kangkaisen closed issue #1124: BE core when load_index

[GitBox]

kangkaisen closed issue #1124: BE core when load_index
URL: https://github.com/apache/incubator-doris/issues/1124
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] kangkaisen opened a new pull request #1818: Add frame_of_reference page

[GitBox]

kangkaisen opened a new pull request #1818: Add frame_of_reference page
URL: https://github.com/apache/incubator-doris/pull/1818
 
 
   For https://github.com/apache/incubator-doris/issues/1817


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] kangkaisen closed issue #1123: BE core dump when ShardedLRUCache::prune

[GitBox]

kangkaisen closed issue #1123: BE core dump when ShardedLRUCache::prune 
URL: https://github.com/apache/incubator-doris/issues/1123
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] morningman commented on a change in pull request #1799: Support table comment and column comment for view

[GitBox]

morningman commented on a change in pull request #1799: Support table comment 
and column comment for view
URL: https://github.com/apache/incubator-doris/pull/1799#discussion_r325004023
 
 

 ##
 File path: docs/documentation/cn/sql-reference/sql-statements/Data 
Definition/CREATE TABLE.md
 ##
 @@ -8,8 +8,9 @@
 [key_desc]
 [partition_desc]
 [distribution_desc]
-[PROPERTIES ("key"="value", ...)];
-[BROKER PROPERTIES ("key"="value", ...)];
+[PROPERTIES ("key"="value", ...)]
+[BROKER PROPERTIES ("key"="value", ...)]
+[COMMENT "table comment"];
 
 Review comment:
   Changed


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] kangkaisen closed issue #1801: Critical bug: Cluster meta will write wrong!

[GitBox]

kangkaisen closed issue #1801: Critical bug: Cluster meta will write wrong!
URL: https://github.com/apache/incubator-doris/issues/1801
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] morningman commented on issue #1814: 这种FE扩容状态是正常的吗

[GitBox]

morningman commented on issue #1814: 这种FE扩容状态是正常的吗
URL: 
https://github.com/apache/incubator-doris/issues/1814#issuecomment-532079194
 
 
   It not right, the `Join` status is false, means those 2 Frontends does not 
join the cluster.
   You need to check fe.log for more information.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[incubator-doris] branch master updated: Limit the max version to cumulative compaction (#1813)

[lichaoyong]

This is an automated email from the ASF dual-hosted git repository.

lichaoyong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git


The following commit(s) were added to refs/heads/master by this push:
 new dc813e6  Limit the max version to cumulative compaction (#1813)
dc813e6 is described below

commit dc813e6c618e97a5a3d6fdf16a3c6fab08565644
Author: lichaoyong 
AuthorDate: Tue Sep 17 14:10:05 2019 +0800

Limit the max version to cumulative compaction (#1813)
---
 be/src/common/config.h| 3 ++-
 be/src/olap/cumulative_compaction.cpp | 8 ++--
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/be/src/common/config.h b/be/src/common/config.h
index f41ee4e..ce5ac46 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -240,7 +240,8 @@ namespace config {
 
 // cumulative compaction policy: max delta file's size unit:B
 CONF_Int32(cumulative_compaction_check_interval_seconds, "10");
-CONF_Int64(cumulative_compaction_num_singleton_deltas, "5");
+CONF_Int64(min_cumulative_compaction_num_singleton_deltas, "5");
+CONF_Int64(max_cumulative_compaction_num_singleton_deltas, "1000");
 CONF_Int32(cumulative_compaction_num_threads, "1");
 CONF_Int32(cumulative_compaction_num_threads_per_disk, "1");
 CONF_Int64(cumulative_compaction_budgeted_bytes, "104857600");
diff --git a/be/src/olap/cumulative_compaction.cpp 
b/be/src/olap/cumulative_compaction.cpp
index 35ddb81..6f8510a 100755
--- a/be/src/olap/cumulative_compaction.cpp
+++ b/be/src/olap/cumulative_compaction.cpp
@@ -85,7 +85,7 @@ OLAPStatus CumulativeCompaction::pick_rowsets_to_compact() {
 // So the ultimate singleton rowset is revserved.
 RowsetSharedPtr rowset = candidate_rowsets[i];
 if (_tablet->version_for_delete_predicate(rowset->version())) {
-if (transient_rowsets.size() > 
config::cumulative_compaction_num_singleton_deltas) {
+if (transient_rowsets.size() > 
config::min_cumulative_compaction_num_singleton_deltas) {
 _input_rowsets = transient_rowsets;
 break;
 }
@@ -93,10 +93,14 @@ OLAPStatus CumulativeCompaction::pick_rowsets_to_compact() {
 continue;
 }
 
+if (transient_rowsets.size() >= 
config::max_cumulative_compaction_num_singleton_deltas) {
+// the threshold of files to compacted one time
+break;
+}
 transient_rowsets.push_back(rowset); 
 }
 
-if (transient_rowsets.size() > 
config::cumulative_compaction_num_singleton_deltas) {
+if (transient_rowsets.size() > 
config::min_cumulative_compaction_num_singleton_deltas) {
 _input_rowsets = transient_rowsets;
 }



-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[GitHub] [incubator-doris] chaoyli merged pull request #1813: Limit the max version to cumulative compaction

[GitBox]

chaoyli merged pull request #1813: Limit the max version to cumulative 
compaction
URL: https://github.com/apache/incubator-doris/pull/1813
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org