Re: [PR] Regexp count [doris]
dwdwqfwe closed pull request #51501: Regexp count URL: https://github.com/apache/doris/pull/51501 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] Regexp count [doris]
dwdwqfwe commented on PR #51501: URL: https://github.com/apache/doris/pull/51501#issuecomment-2960849482 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] Regexp count [doris]
dwdwqfwe commented on PR #51501: URL: https://github.com/apache/doris/pull/51501#issuecomment-2960826151 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] Regexp count [doris]
dwdwqfwe commented on code in PR #51501:
URL: https://github.com/apache/doris/pull/51501#discussion_r2138045798
##
be/src/vec/functions/function_regexp.cpp:
##
@@ -50,6 +52,152 @@
namespace doris::vectorized {
#include "common/compile_check_begin.h"
+struct RegexpCountImpl {
+static void execute_impl(FunctionContext* context, ColumnPtr
argument_columns[],
+ size_t input_rows_count, ColumnInt64::Container&
result_data,
+ NullMap& null_map) {
+const auto* pattern =
check_and_get_column(argument_columns[1].get());
+const auto* str =
check_and_get_column(argument_columns[0].get());
+
+for (size_t i = 0; i < input_rows_count; ++i) {
+if (null_map[i]) {
+result_data[i] = 0;
+continue;
+}
+
+result_data[i] = _execute_inner_loop(context, str, pattern,
null_map, i);
+}
+}
+
+private:
+static int64_t _execute_inner_loop(FunctionContext* context, const
ColumnString* str,
+ const ColumnString* pattern, NullMap&
null_map,
+ const size_t index_now) {
+re2::RE2* re = reinterpret_cast(
+context->get_function_state(FunctionContext::THREAD_LOCAL));
+std::unique_ptr scoped_re;
Review Comment:
I used scoped_re int here "bool st = StringFunctions::compile_regex(pattern,
&error_str, StringRef(), StringRef(),
scoped_re);"
and I have checked re is not nullptr at "if (!re) {
std::string error_str;"
##
be/src/vec/functions/function_regexp.cpp:
##
@@ -50,6 +52,152 @@
namespace doris::vectorized {
#include "common/compile_check_begin.h"
+struct RegexpCountImpl {
+static void execute_impl(FunctionContext* context, ColumnPtr
argument_columns[],
+ size_t input_rows_count, ColumnInt64::Container&
result_data,
+ NullMap& null_map) {
+const auto* pattern =
check_and_get_column(argument_columns[1].get());
+const auto* str =
check_and_get_column(argument_columns[0].get());
+
+for (size_t i = 0; i < input_rows_count; ++i) {
+if (null_map[i]) {
+result_data[i] = 0;
+continue;
+}
+
+result_data[i] = _execute_inner_loop(context, str, pattern,
null_map, i);
+}
+}
+
+private:
+static int64_t _execute_inner_loop(FunctionContext* context, const
ColumnString* str,
+ const ColumnString* pattern, NullMap&
null_map,
+ const size_t index_now) {
+re2::RE2* re = reinterpret_cast(
+context->get_function_state(FunctionContext::THREAD_LOCAL));
+std::unique_ptr scoped_re;
+
+if (str->is_null_at(index_now) || pattern->is_null_at(index_now)) {
+null_map[index_now] = true;
+return 0;
+}
+
+const auto& str_data = str->get_data_at(index_now);
+const auto& pattern_data = pattern->get_data_at(index_now);
+
+if (!re) {
+std::string error_str;
+bool st = StringFunctions::compile_regex(pattern_data, &error_str,
StringRef(),
+ StringRef(), scoped_re);
+if (!st) {
+context->add_warning(error_str.c_str());
+null_map[index_now] = true;
+return 0;
+}
+re = scoped_re.get();
+}
+
+int64_t count = 0;
+size_t pos = 0;
+re2::StringPiece str_sp(str_data.data, str_data.size);
+
+while (pos < str_data.size) {
+re2::StringPiece current(str_data.data + pos, str_data.size - pos);
+re2::StringPiece match;
+
+if (!re->Match(current, 0, current.size(), re2::RE2::UNANCHORED,
&match, 1)) {
+break;
+}
+
+if (match.empty()) {
+pos++;
+} else {
+count++;
+pos += match.data() - current.data() + match.size();
+}
+}
+
+return count;
+}
+};
+
+class FunctionRegexpCount : public IFunction {
+public:
+static constexpr auto name = "regexp_count";
+
+static FunctionPtr create() { return
std::make_shared(); }
+
+String get_name() const override { return name; }
+
+size_t get_number_of_arguments() const override { return 2; }
+
+DataTypePtr get_return_type_impl(const DataTypes& arguments) const
override {
+auto int64_type = std::make_shared();
+return make_nullable(std::static_pointer_cast(int64_type));
+}
+
+Status open(FunctionContext* context, FunctionContext::FunctionStateScope
scope) override {
+if (scope == FunctionContext::THREAD_LOCAL &&
Re: [PR] Regexp count [doris]
dwdwqfwe commented on PR #51501: URL: https://github.com/apache/doris/pull/51501#issuecomment-2960822553 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] Regexp count [doris]
dwdwqfwe commented on PR #51501: URL: https://github.com/apache/doris/pull/51501#issuecomment-2960805856 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] Regexp count [doris]
zhangstar333 commented on PR #51501: URL: https://github.com/apache/doris/pull/51501#issuecomment-2958918563 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] Regexp count [doris]
zhangstar333 commented on code in PR #51501:
URL: https://github.com/apache/doris/pull/51501#discussion_r2137407554
##
be/src/vec/functions/function_regexp.cpp:
##
@@ -50,6 +52,152 @@
namespace doris::vectorized {
#include "common/compile_check_begin.h"
+struct RegexpCountImpl {
+static void execute_impl(FunctionContext* context, ColumnPtr
argument_columns[],
+ size_t input_rows_count, ColumnInt64::Container&
result_data,
+ NullMap& null_map) {
+const auto* pattern =
check_and_get_column(argument_columns[1].get());
+const auto* str =
check_and_get_column(argument_columns[0].get());
+
+for (size_t i = 0; i < input_rows_count; ++i) {
+if (null_map[i]) {
+result_data[i] = 0;
+continue;
+}
+
+result_data[i] = _execute_inner_loop(context, str, pattern,
null_map, i);
+}
+}
+
+private:
+static int64_t _execute_inner_loop(FunctionContext* context, const
ColumnString* str,
+ const ColumnString* pattern, NullMap&
null_map,
+ const size_t index_now) {
+re2::RE2* re = reinterpret_cast(
+context->get_function_state(FunctionContext::THREAD_LOCAL));
+std::unique_ptr scoped_re;
+
+if (str->is_null_at(index_now) || pattern->is_null_at(index_now)) {
+null_map[index_now] = true;
+return 0;
+}
+
+const auto& str_data = str->get_data_at(index_now);
+const auto& pattern_data = pattern->get_data_at(index_now);
+
+if (!re) {
+std::string error_str;
+bool st = StringFunctions::compile_regex(pattern_data, &error_str,
StringRef(),
+ StringRef(), scoped_re);
+if (!st) {
+context->add_warning(error_str.c_str());
+null_map[index_now] = true;
+return 0;
+}
+re = scoped_re.get();
+}
+
+int64_t count = 0;
+size_t pos = 0;
+re2::StringPiece str_sp(str_data.data, str_data.size);
+
+while (pos < str_data.size) {
+re2::StringPiece current(str_data.data + pos, str_data.size - pos);
+re2::StringPiece match;
+
+if (!re->Match(current, 0, current.size(), re2::RE2::UNANCHORED,
&match, 1)) {
+break;
+}
+
+if (match.empty()) {
+pos++;
+} else {
+count++;
+pos += match.data() - current.data() + match.size();
Review Comment:
could add some docment about
pos += match.data() - current.data() + match.size();
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Regexp count [doris]
zhangstar333 commented on code in PR #51501:
URL: https://github.com/apache/doris/pull/51501#discussion_r2137400762
##
be/src/vec/functions/function_regexp.cpp:
##
@@ -50,6 +52,152 @@
namespace doris::vectorized {
#include "common/compile_check_begin.h"
+struct RegexpCountImpl {
+static void execute_impl(FunctionContext* context, ColumnPtr
argument_columns[],
+ size_t input_rows_count, ColumnInt64::Container&
result_data,
+ NullMap& null_map) {
+const auto* pattern =
check_and_get_column(argument_columns[1].get());
+const auto* str =
check_and_get_column(argument_columns[0].get());
+
+for (size_t i = 0; i < input_rows_count; ++i) {
+if (null_map[i]) {
+result_data[i] = 0;
+continue;
+}
+
+result_data[i] = _execute_inner_loop(context, str, pattern,
null_map, i);
+}
+}
+
+private:
+static int64_t _execute_inner_loop(FunctionContext* context, const
ColumnString* str,
+ const ColumnString* pattern, NullMap&
null_map,
+ const size_t index_now) {
+re2::RE2* re = reinterpret_cast(
+context->get_function_state(FunctionContext::THREAD_LOCAL));
+std::unique_ptr scoped_re;
+
+if (str->is_null_at(index_now) || pattern->is_null_at(index_now)) {
+null_map[index_now] = true;
+return 0;
+}
+
+const auto& str_data = str->get_data_at(index_now);
+const auto& pattern_data = pattern->get_data_at(index_now);
+
+if (!re) {
+std::string error_str;
+bool st = StringFunctions::compile_regex(pattern_data, &error_str,
StringRef(),
+ StringRef(), scoped_re);
+if (!st) {
+context->add_warning(error_str.c_str());
+null_map[index_now] = true;
+return 0;
+}
+re = scoped_re.get();
+}
+
+int64_t count = 0;
+size_t pos = 0;
+re2::StringPiece str_sp(str_data.data, str_data.size);
Review Comment:
not used str_sp?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Regexp count [doris]
zhangstar333 commented on code in PR #51501:
URL: https://github.com/apache/doris/pull/51501#discussion_r2137397490
##
be/src/vec/functions/function_regexp.cpp:
##
@@ -50,6 +52,152 @@
namespace doris::vectorized {
#include "common/compile_check_begin.h"
+struct RegexpCountImpl {
+static void execute_impl(FunctionContext* context, ColumnPtr
argument_columns[],
+ size_t input_rows_count, ColumnInt64::Container&
result_data,
+ NullMap& null_map) {
+const auto* pattern =
check_and_get_column(argument_columns[1].get());
+const auto* str =
check_and_get_column(argument_columns[0].get());
+
+for (size_t i = 0; i < input_rows_count; ++i) {
+if (null_map[i]) {
+result_data[i] = 0;
+continue;
+}
+
+result_data[i] = _execute_inner_loop(context, str, pattern,
null_map, i);
+}
+}
+
+private:
+static int64_t _execute_inner_loop(FunctionContext* context, const
ColumnString* str,
+ const ColumnString* pattern, NullMap&
null_map,
+ const size_t index_now) {
+re2::RE2* re = reinterpret_cast(
+context->get_function_state(FunctionContext::THREAD_LOCAL));
+std::unique_ptr scoped_re;
Review Comment:
not used scoped_re?
and maybe could add check re is not nullptr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Regexp count [doris]
zhangstar333 commented on code in PR #51501:
URL: https://github.com/apache/doris/pull/51501#discussion_r2137212293
##
be/src/vec/functions/function_regexp.cpp:
##
@@ -50,6 +52,152 @@
namespace doris::vectorized {
#include "common/compile_check_begin.h"
+struct RegexpCountImpl {
+static void execute_impl(FunctionContext* context, ColumnPtr
argument_columns[],
+ size_t input_rows_count, ColumnInt64::Container&
result_data,
+ NullMap& null_map) {
+const auto* pattern =
check_and_get_column(argument_columns[1].get());
+const auto* str =
check_and_get_column(argument_columns[0].get());
+
+for (size_t i = 0; i < input_rows_count; ++i) {
+if (null_map[i]) {
+result_data[i] = 0;
+continue;
+}
+
+result_data[i] = _execute_inner_loop(context, str, pattern,
null_map, i);
+}
+}
+
+private:
+static int64_t _execute_inner_loop(FunctionContext* context, const
ColumnString* str,
+ const ColumnString* pattern, NullMap&
null_map,
+ const size_t index_now) {
+re2::RE2* re = reinterpret_cast(
+context->get_function_state(FunctionContext::THREAD_LOCAL));
+std::unique_ptr scoped_re;
+
+if (str->is_null_at(index_now) || pattern->is_null_at(index_now)) {
+null_map[index_now] = true;
+return 0;
+}
+
+const auto& str_data = str->get_data_at(index_now);
+const auto& pattern_data = pattern->get_data_at(index_now);
+
+if (!re) {
+std::string error_str;
+bool st = StringFunctions::compile_regex(pattern_data, &error_str,
StringRef(),
+ StringRef(), scoped_re);
+if (!st) {
+context->add_warning(error_str.c_str());
+null_map[index_now] = true;
+return 0;
+}
+re = scoped_re.get();
+}
+
+int64_t count = 0;
+size_t pos = 0;
+re2::StringPiece str_sp(str_data.data, str_data.size);
+
+while (pos < str_data.size) {
+re2::StringPiece current(str_data.data + pos, str_data.size - pos);
+re2::StringPiece match;
+
+if (!re->Match(current, 0, current.size(), re2::RE2::UNANCHORED,
&match, 1)) {
+break;
+}
+
+if (match.empty()) {
+pos++;
+} else {
+count++;
+pos += match.data() - current.data() + match.size();
+}
+}
+
+return count;
+}
+};
+
+class FunctionRegexpCount : public IFunction {
+public:
+static constexpr auto name = "regexp_count";
+
+static FunctionPtr create() { return
std::make_shared(); }
+
+String get_name() const override { return name; }
+
+size_t get_number_of_arguments() const override { return 2; }
+
+DataTypePtr get_return_type_impl(const DataTypes& arguments) const
override {
+auto int64_type = std::make_shared();
+return make_nullable(std::static_pointer_cast(int64_type));
Review Comment:
could return make_nullable(std::make_shared());
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Regexp count [doris]
zhangstar333 commented on code in PR #51501:
URL: https://github.com/apache/doris/pull/51501#discussion_r2137233781
##
be/src/vec/functions/function_regexp.cpp:
##
@@ -50,6 +52,152 @@
namespace doris::vectorized {
#include "common/compile_check_begin.h"
+struct RegexpCountImpl {
+static void execute_impl(FunctionContext* context, ColumnPtr
argument_columns[],
+ size_t input_rows_count, ColumnInt64::Container&
result_data,
+ NullMap& null_map) {
+const auto* pattern =
check_and_get_column(argument_columns[1].get());
+const auto* str =
check_and_get_column(argument_columns[0].get());
+
+for (size_t i = 0; i < input_rows_count; ++i) {
+if (null_map[i]) {
+result_data[i] = 0;
+continue;
+}
+
+result_data[i] = _execute_inner_loop(context, str, pattern,
null_map, i);
+}
+}
+
+private:
+static int64_t _execute_inner_loop(FunctionContext* context, const
ColumnString* str,
+ const ColumnString* pattern, NullMap&
null_map,
+ const size_t index_now) {
+re2::RE2* re = reinterpret_cast(
+context->get_function_state(FunctionContext::THREAD_LOCAL));
+std::unique_ptr scoped_re;
+
+if (str->is_null_at(index_now) || pattern->is_null_at(index_now)) {
+null_map[index_now] = true;
+return 0;
+}
+
+const auto& str_data = str->get_data_at(index_now);
+const auto& pattern_data = pattern->get_data_at(index_now);
+
+if (!re) {
+std::string error_str;
+bool st = StringFunctions::compile_regex(pattern_data, &error_str,
StringRef(),
+ StringRef(), scoped_re);
+if (!st) {
+context->add_warning(error_str.c_str());
+null_map[index_now] = true;
+return 0;
+}
+re = scoped_re.get();
+}
+
+int64_t count = 0;
+size_t pos = 0;
+re2::StringPiece str_sp(str_data.data, str_data.size);
+
+while (pos < str_data.size) {
+re2::StringPiece current(str_data.data + pos, str_data.size - pos);
+re2::StringPiece match;
+
+if (!re->Match(current, 0, current.size(), re2::RE2::UNANCHORED,
&match, 1)) {
+break;
+}
+
+if (match.empty()) {
+pos++;
+} else {
+count++;
+pos += match.data() - current.data() + match.size();
+}
+}
+
+return count;
+}
+};
+
+class FunctionRegexpCount : public IFunction {
+public:
+static constexpr auto name = "regexp_count";
+
+static FunctionPtr create() { return
std::make_shared(); }
+
+String get_name() const override { return name; }
+
+size_t get_number_of_arguments() const override { return 2; }
+
+DataTypePtr get_return_type_impl(const DataTypes& arguments) const
override {
+auto int64_type = std::make_shared();
+return make_nullable(std::static_pointer_cast(int64_type));
+}
+
+Status open(FunctionContext* context, FunctionContext::FunctionStateScope
scope) override {
+if (scope == FunctionContext::THREAD_LOCAL &&
context->is_col_constant(1)) {
+const auto pattern_col = context->get_constant_col(1)->column_ptr;
+const auto& pattern = pattern_col->get_data_at(0);
+if (pattern.size == 0) {
+return Status::OK();
+}
+std::string error_str;
+std::unique_ptr scoped_re;
+bool st = StringFunctions::compile_regex(pattern, &error_str,
StringRef(), StringRef(),
+ scoped_re);
+if (!st) {
+context->set_error(error_str.c_str());
+return Status::InvalidArgument(error_str);
+}
+std::shared_ptr re(scoped_re.release());
+context->set_function_state(scope,
std::static_pointer_cast(re));
+}
+return Status::OK();
+}
+
+Status close(FunctionContext* context, FunctionContext::FunctionStateScope
scope) override {
+if (scope == FunctionContext::THREAD_LOCAL) {
+auto ptr = context->get_function_state(scope);
+if (ptr) {
+delete reinterpret_cast(ptr);
Review Comment:
seeme you set is std::shared_ptr, it's need the delete?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
Re: [PR] Regexp count [doris]
zhangstar333 commented on code in PR #51501:
URL: https://github.com/apache/doris/pull/51501#discussion_r2137240776
##
be/src/vec/functions/function_regexp.cpp:
##
@@ -50,6 +52,152 @@
namespace doris::vectorized {
#include "common/compile_check_begin.h"
+struct RegexpCountImpl {
+static void execute_impl(FunctionContext* context, ColumnPtr
argument_columns[],
+ size_t input_rows_count, ColumnInt64::Container&
result_data,
+ NullMap& null_map) {
+const auto* pattern =
check_and_get_column(argument_columns[1].get());
+const auto* str =
check_and_get_column(argument_columns[0].get());
+
+for (size_t i = 0; i < input_rows_count; ++i) {
+if (null_map[i]) {
+result_data[i] = 0;
+continue;
+}
+
+result_data[i] = _execute_inner_loop(context, str, pattern,
null_map, i);
+}
+}
+
+private:
+static int64_t _execute_inner_loop(FunctionContext* context, const
ColumnString* str,
+ const ColumnString* pattern, NullMap&
null_map,
+ const size_t index_now) {
+re2::RE2* re = reinterpret_cast(
+context->get_function_state(FunctionContext::THREAD_LOCAL));
+std::unique_ptr scoped_re;
+
+if (str->is_null_at(index_now) || pattern->is_null_at(index_now)) {
+null_map[index_now] = true;
+return 0;
+}
+
+const auto& str_data = str->get_data_at(index_now);
+const auto& pattern_data = pattern->get_data_at(index_now);
+
+if (!re) {
+std::string error_str;
+bool st = StringFunctions::compile_regex(pattern_data, &error_str,
StringRef(),
+ StringRef(), scoped_re);
+if (!st) {
+context->add_warning(error_str.c_str());
+null_map[index_now] = true;
+return 0;
+}
+re = scoped_re.get();
+}
+
+int64_t count = 0;
+size_t pos = 0;
+re2::StringPiece str_sp(str_data.data, str_data.size);
+
+while (pos < str_data.size) {
+re2::StringPiece current(str_data.data + pos, str_data.size - pos);
+re2::StringPiece match;
+
+if (!re->Match(current, 0, current.size(), re2::RE2::UNANCHORED,
&match, 1)) {
+break;
+}
+
+if (match.empty()) {
+pos++;
+} else {
+count++;
+pos += match.data() - current.data() + match.size();
+}
+}
+
+return count;
+}
+};
+
+class FunctionRegexpCount : public IFunction {
+public:
+static constexpr auto name = "regexp_count";
+
+static FunctionPtr create() { return
std::make_shared(); }
+
+String get_name() const override { return name; }
+
+size_t get_number_of_arguments() const override { return 2; }
+
+DataTypePtr get_return_type_impl(const DataTypes& arguments) const
override {
+auto int64_type = std::make_shared();
+return make_nullable(std::static_pointer_cast(int64_type));
+}
+
+Status open(FunctionContext* context, FunctionContext::FunctionStateScope
scope) override {
+if (scope == FunctionContext::THREAD_LOCAL &&
context->is_col_constant(1)) {
+const auto pattern_col = context->get_constant_col(1)->column_ptr;
+const auto& pattern = pattern_col->get_data_at(0);
+if (pattern.size == 0) {
+return Status::OK();
+}
+std::string error_str;
+std::unique_ptr scoped_re;
+bool st = StringFunctions::compile_regex(pattern, &error_str,
StringRef(), StringRef(),
+ scoped_re);
+if (!st) {
+context->set_error(error_str.c_str());
+return Status::InvalidArgument(error_str);
+}
+std::shared_ptr re(scoped_re.release());
+context->set_function_state(scope,
std::static_pointer_cast(re));
+}
+return Status::OK();
+}
+
+Status close(FunctionContext* context, FunctionContext::FunctionStateScope
scope) override {
+if (scope == FunctionContext::THREAD_LOCAL) {
+auto ptr = context->get_function_state(scope);
+if (ptr) {
+delete reinterpret_cast(ptr);
+context->set_function_state(scope, nullptr);
+}
+}
+return Status::OK();
+}
+
+Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
+uint32_t result, size_t input_rows_count) const
override {
+auto result_null_map = ColumnUInt8::create(input_rows_count, 0);
+auto result_data_column = ColumnInt
Re: [PR] Regexp count [doris]
zhangstar333 commented on code in PR #51501:
URL: https://github.com/apache/doris/pull/51501#discussion_r2137247465
##
be/src/vec/functions/function_regexp.cpp:
##
@@ -50,6 +52,152 @@
namespace doris::vectorized {
#include "common/compile_check_begin.h"
+struct RegexpCountImpl {
+static void execute_impl(FunctionContext* context, ColumnPtr
argument_columns[],
+ size_t input_rows_count, ColumnInt64::Container&
result_data,
+ NullMap& null_map) {
+const auto* pattern =
check_and_get_column(argument_columns[1].get());
+const auto* str =
check_and_get_column(argument_columns[0].get());
+
+for (size_t i = 0; i < input_rows_count; ++i) {
+if (null_map[i]) {
Review Comment:
the check is used for ?? you null_map seems is result_null_map->get_data()
and the null_map is init value is 0,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Regexp count [doris]
zhangstar333 commented on code in PR #51501:
URL: https://github.com/apache/doris/pull/51501#discussion_r2137235469
##
be/src/vec/functions/function_regexp.cpp:
##
@@ -50,6 +52,152 @@
namespace doris::vectorized {
#include "common/compile_check_begin.h"
+struct RegexpCountImpl {
+static void execute_impl(FunctionContext* context, ColumnPtr
argument_columns[],
+ size_t input_rows_count, ColumnInt64::Container&
result_data,
+ NullMap& null_map) {
+const auto* pattern =
check_and_get_column(argument_columns[1].get());
+const auto* str =
check_and_get_column(argument_columns[0].get());
+
+for (size_t i = 0; i < input_rows_count; ++i) {
+if (null_map[i]) {
+result_data[i] = 0;
+continue;
+}
+
+result_data[i] = _execute_inner_loop(context, str, pattern,
null_map, i);
+}
+}
+
+private:
+static int64_t _execute_inner_loop(FunctionContext* context, const
ColumnString* str,
+ const ColumnString* pattern, NullMap&
null_map,
+ const size_t index_now) {
+re2::RE2* re = reinterpret_cast(
+context->get_function_state(FunctionContext::THREAD_LOCAL));
+std::unique_ptr scoped_re;
+
+if (str->is_null_at(index_now) || pattern->is_null_at(index_now)) {
+null_map[index_now] = true;
+return 0;
+}
+
+const auto& str_data = str->get_data_at(index_now);
+const auto& pattern_data = pattern->get_data_at(index_now);
+
+if (!re) {
+std::string error_str;
+bool st = StringFunctions::compile_regex(pattern_data, &error_str,
StringRef(),
+ StringRef(), scoped_re);
+if (!st) {
+context->add_warning(error_str.c_str());
+null_map[index_now] = true;
+return 0;
+}
+re = scoped_re.get();
+}
+
+int64_t count = 0;
+size_t pos = 0;
+re2::StringPiece str_sp(str_data.data, str_data.size);
+
+while (pos < str_data.size) {
+re2::StringPiece current(str_data.data + pos, str_data.size - pos);
+re2::StringPiece match;
+
+if (!re->Match(current, 0, current.size(), re2::RE2::UNANCHORED,
&match, 1)) {
+break;
+}
+
+if (match.empty()) {
+pos++;
+} else {
+count++;
+pos += match.data() - current.data() + match.size();
+}
+}
+
+return count;
+}
+};
+
+class FunctionRegexpCount : public IFunction {
+public:
+static constexpr auto name = "regexp_count";
+
+static FunctionPtr create() { return
std::make_shared(); }
+
+String get_name() const override { return name; }
+
+size_t get_number_of_arguments() const override { return 2; }
+
+DataTypePtr get_return_type_impl(const DataTypes& arguments) const
override {
+auto int64_type = std::make_shared();
+return make_nullable(std::static_pointer_cast(int64_type));
+}
+
+Status open(FunctionContext* context, FunctionContext::FunctionStateScope
scope) override {
+if (scope == FunctionContext::THREAD_LOCAL &&
context->is_col_constant(1)) {
+const auto pattern_col = context->get_constant_col(1)->column_ptr;
+const auto& pattern = pattern_col->get_data_at(0);
+if (pattern.size == 0) {
+return Status::OK();
+}
+std::string error_str;
+std::unique_ptr scoped_re;
+bool st = StringFunctions::compile_regex(pattern, &error_str,
StringRef(), StringRef(),
+ scoped_re);
+if (!st) {
+context->set_error(error_str.c_str());
+return Status::InvalidArgument(error_str);
+}
+std::shared_ptr re(scoped_re.release());
+context->set_function_state(scope,
std::static_pointer_cast(re));
+}
+return Status::OK();
+}
+
+Status close(FunctionContext* context, FunctionContext::FunctionStateScope
scope) override {
+if (scope == FunctionContext::THREAD_LOCAL) {
+auto ptr = context->get_function_state(scope);
+if (ptr) {
+delete reinterpret_cast(ptr);
+context->set_function_state(scope, nullptr);
+}
+}
+return Status::OK();
+}
+
+Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
+uint32_t result, size_t input_rows_count) const
override {
+auto result_null_map = ColumnUInt8::create(input_rows_count, 0);
+auto result_data_column = ColumnInt
Re: [PR] Regexp count [doris]
zhangstar333 commented on code in PR #51501:
URL: https://github.com/apache/doris/pull/51501#discussion_r2137240776
##
be/src/vec/functions/function_regexp.cpp:
##
@@ -50,6 +52,152 @@
namespace doris::vectorized {
#include "common/compile_check_begin.h"
+struct RegexpCountImpl {
+static void execute_impl(FunctionContext* context, ColumnPtr
argument_columns[],
+ size_t input_rows_count, ColumnInt64::Container&
result_data,
+ NullMap& null_map) {
+const auto* pattern =
check_and_get_column(argument_columns[1].get());
+const auto* str =
check_and_get_column(argument_columns[0].get());
+
+for (size_t i = 0; i < input_rows_count; ++i) {
+if (null_map[i]) {
+result_data[i] = 0;
+continue;
+}
+
+result_data[i] = _execute_inner_loop(context, str, pattern,
null_map, i);
+}
+}
+
+private:
+static int64_t _execute_inner_loop(FunctionContext* context, const
ColumnString* str,
+ const ColumnString* pattern, NullMap&
null_map,
+ const size_t index_now) {
+re2::RE2* re = reinterpret_cast(
+context->get_function_state(FunctionContext::THREAD_LOCAL));
+std::unique_ptr scoped_re;
+
+if (str->is_null_at(index_now) || pattern->is_null_at(index_now)) {
+null_map[index_now] = true;
+return 0;
+}
+
+const auto& str_data = str->get_data_at(index_now);
+const auto& pattern_data = pattern->get_data_at(index_now);
+
+if (!re) {
+std::string error_str;
+bool st = StringFunctions::compile_regex(pattern_data, &error_str,
StringRef(),
+ StringRef(), scoped_re);
+if (!st) {
+context->add_warning(error_str.c_str());
+null_map[index_now] = true;
+return 0;
+}
+re = scoped_re.get();
+}
+
+int64_t count = 0;
+size_t pos = 0;
+re2::StringPiece str_sp(str_data.data, str_data.size);
+
+while (pos < str_data.size) {
+re2::StringPiece current(str_data.data + pos, str_data.size - pos);
+re2::StringPiece match;
+
+if (!re->Match(current, 0, current.size(), re2::RE2::UNANCHORED,
&match, 1)) {
+break;
+}
+
+if (match.empty()) {
+pos++;
+} else {
+count++;
+pos += match.data() - current.data() + match.size();
+}
+}
+
+return count;
+}
+};
+
+class FunctionRegexpCount : public IFunction {
+public:
+static constexpr auto name = "regexp_count";
+
+static FunctionPtr create() { return
std::make_shared(); }
+
+String get_name() const override { return name; }
+
+size_t get_number_of_arguments() const override { return 2; }
+
+DataTypePtr get_return_type_impl(const DataTypes& arguments) const
override {
+auto int64_type = std::make_shared();
+return make_nullable(std::static_pointer_cast(int64_type));
+}
+
+Status open(FunctionContext* context, FunctionContext::FunctionStateScope
scope) override {
+if (scope == FunctionContext::THREAD_LOCAL &&
context->is_col_constant(1)) {
+const auto pattern_col = context->get_constant_col(1)->column_ptr;
+const auto& pattern = pattern_col->get_data_at(0);
+if (pattern.size == 0) {
+return Status::OK();
+}
+std::string error_str;
+std::unique_ptr scoped_re;
+bool st = StringFunctions::compile_regex(pattern, &error_str,
StringRef(), StringRef(),
+ scoped_re);
+if (!st) {
+context->set_error(error_str.c_str());
+return Status::InvalidArgument(error_str);
+}
+std::shared_ptr re(scoped_re.release());
+context->set_function_state(scope,
std::static_pointer_cast(re));
+}
+return Status::OK();
+}
+
+Status close(FunctionContext* context, FunctionContext::FunctionStateScope
scope) override {
+if (scope == FunctionContext::THREAD_LOCAL) {
+auto ptr = context->get_function_state(scope);
+if (ptr) {
+delete reinterpret_cast(ptr);
+context->set_function_state(scope, nullptr);
+}
+}
+return Status::OK();
+}
+
+Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
+uint32_t result, size_t input_rows_count) const
override {
+auto result_null_map = ColumnUInt8::create(input_rows_count, 0);
+auto result_data_column = ColumnInt
Re: [PR] Regexp count [doris]
zhangstar333 commented on code in PR #51501:
URL: https://github.com/apache/doris/pull/51501#discussion_r2137247465
##
be/src/vec/functions/function_regexp.cpp:
##
@@ -50,6 +52,152 @@
namespace doris::vectorized {
#include "common/compile_check_begin.h"
+struct RegexpCountImpl {
+static void execute_impl(FunctionContext* context, ColumnPtr
argument_columns[],
+ size_t input_rows_count, ColumnInt64::Container&
result_data,
+ NullMap& null_map) {
+const auto* pattern =
check_and_get_column(argument_columns[1].get());
+const auto* str =
check_and_get_column(argument_columns[0].get());
+
+for (size_t i = 0; i < input_rows_count; ++i) {
+if (null_map[i]) {
Review Comment:
the check is used for ??
and the null_map is init value is 0,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Regexp count [doris]
hello-stephen commented on PR #51501: URL: https://github.com/apache/doris/pull/51501#issuecomment-2941921984 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
