rkavanap commented on a change in pull request #11052:
URL: https://github.com/apache/arrow/pull/11052#discussion_r740730066
##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -2195,4 +2195,74 @@ const char* byte_substr_binary_int32_int32(gdv_int64
context, const char* text,
memcpy(ret, text + startPos, *out_len);
return ret;
}
+
+// Array that maps each letter from the alphabet to its corresponding number
for the soundex algorithm.
+// ABCDEFGHIJKLMNOPQRSTUVWXYZ -> 01230120022455012623010202
+static char mappings[] =
{'0','1','2','3','0','1','2','0','0','2','2','4','5','5','0','1','2','6','2','3','0','1','0','2','0','2'};
+
+// Returns the soundex code for a given string
+//
+// The soundex function evaluates expression and returns the most significant
letter in
+// the input string followed by a phonetic code. Characters that are not
alphabetic are
+// ignored. If expression evaluates to the null value, null is returned.
+//
+// The soundex algorith works with the following steps:
+// 1. Retain the first letter of the string and drop all other occurrences
of a, e, i, o, u, y, h, w.
+// 2. Replace consonants with digits as follows (after the first letter):
+// b, f, p, v → 1
+// c, g, j, k, q, s, x, z → 2
+// d, t → 3
+// l → 4
+// m, n → 5
+// r → 6
+// 3. If two or more letters with the same number are adjacent in the
original name
+// (before step 1), only retain the first letter; also two letters with the
same number
+// separated by 'h' or 'w' are coded as a single number, whereas such
letters separated
+// by a vowel are coded twice. This rule also applies to the first letter.
+// 4. If the string have too few letters in the word that you can't assign
three numbers,
+// append with zeros until there are three numbers. If you have four or
more numbers,
+// retain only the first three.
+FORCE_INLINE
+const char* soundex_utf8(gdv_int64 ctx, const char* in, gdv_int32 in_len,
+ int32_t* out_len) {
+ if (in_len <= 0) {
+ *out_len = 0;
+ return "";
+ }
+
+ // The soundex code is composed by one letter and three numbers
+ char* ret = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(ctx, 4));
+ if (ret == nullptr) {
+ gdv_fn_context_set_error_msg(ctx, "Could not allocate memory for output
string");
+ *out_len = 0;
+ return "";
+ }
+
+ int si = 1;
+ unsigned char c;
+ ret[0] = toupper(in[0]);
Review comment:
what if the first character is not a letter?. Shouldn't we skip it to
run the algorithm? e.g "1SMITH" or "123SMYTHE"
##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -2195,4 +2195,74 @@ const char* byte_substr_binary_int32_int32(gdv_int64
context, const char* text,
memcpy(ret, text + startPos, *out_len);
return ret;
}
+
+// Array that maps each letter from the alphabet to its corresponding number
for the soundex algorithm.
+// ABCDEFGHIJKLMNOPQRSTUVWXYZ -> 01230120022455012623010202
+static char mappings[] =
{'0','1','2','3','0','1','2','0','0','2','2','4','5','5','0','1','2','6','2','3','0','1','0','2','0','2'};
+
+// Returns the soundex code for a given string
+//
+// The soundex function evaluates expression and returns the most significant
letter in
+// the input string followed by a phonetic code. Characters that are not
alphabetic are
+// ignored. If expression evaluates to the null value, null is returned.
+//
+// The soundex algorith works with the following steps:
+// 1. Retain the first letter of the string and drop all other occurrences
of a, e, i, o, u, y, h, w.
+// 2. Replace consonants with digits as follows (after the first letter):
+// b, f, p, v → 1
+// c, g, j, k, q, s, x, z → 2
+// d, t → 3
+// l → 4
+// m, n → 5
+// r → 6
+// 3. If two or more letters with the same number are adjacent in the
original name
+// (before step 1), only retain the first letter; also two letters with the
same number
+// separated by 'h' or 'w' are coded as a single number, whereas such
letters separated
+// by a vowel are coded twice. This rule also applies to the first letter.
+// 4. If the string have too few letters in the word that you can't assign
three numbers,
+// append with zeros until there are three numbers. If you have four or
more numbers,
+// retain only the first three.
+FORCE_INLINE
+const char* soundex_utf8(gdv_int64 ctx, const char* in, gdv_int32 in_len,
+ int32_t* out_len) {
+ if (in_len <= 0) {
+ *out_len = 0;
+ return "";
+ }
+
+ // The soundex code is composed by one letter and three numbers
+ char* ret = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(ctx, 4));
+ if (ret == nullptr) {
+ gdv_fn_context_set_error_msg(ctx, "Could not allocate memory for output
string");
+ *out_len = 0;
+ return "";
+ }
+
+ int si = 1;
+ unsigned char c;
+ ret[0] = toupper(in[0]);
+ for(int i = 1, l = in_len; i < l; i++) {
+ c = toupper(in[i]) - 65;
Review comment:
just wondering if it is faster to check isAlpha (basically remove if in
2247 and do lines 2245 to 2256 only if it is a letter)
##########
File path: cpp/src/gandiva/precompiled/string_ops_test.cc
##########
@@ -1755,4 +1755,23 @@ TEST(TestStringOps, TestConvertToBigEndian) {
#endif
}
+TEST(TestStringOps, TestSoundex) {
+ gandiva::ExecutionContext ctx;
+ auto ctx_ptr = reinterpret_cast<int64_t>(&ctx);
+ int32_t out_len = 0;
+ const char* out;
+
+ out = soundex_utf8(ctx_ptr, "Miller", 6, &out_len);
+ EXPECT_EQ(std::string(out, out_len), "M460");
+
+ out = soundex_utf8(ctx_ptr, "abc", 3, &out_len);
+ EXPECT_EQ(std::string(out, out_len), "A120");
+
+ out = soundex_utf8(ctx_ptr, "test", 4, &out_len);
+ EXPECT_EQ(std::string(out, out_len), "T230");
Review comment:
A test case having non letter at the beginning and vowels in the
beginning..?
"3Miller", "Elvis", "waterloo" ,"eowolf"
also a test case that compares the output of 2 similar sounding values. e.g
Smith and Smythe are similar sounding and probably should return same value?.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
