mapleFU commented on code in PR #55408:
URL: https://github.com/apache/arrow/pull/55408#discussion_r4586687306


##########
cpp/src/parquet/level_comparison.h:
##########
@@ -35,4 +36,78 @@ struct MinMax {
 
 MinMax PARQUET_EXPORT FindMinMax(const int16_t* levels, int64_t num_levels);
 
+/// Counts the levels that satisfy `pred`, in chunks small enough that a
+/// `uint16_t` accumulator cannot overflow.
+///
+/// This is the fallback for inputs that do not fit in a `uint16_t` 
accumulator.
+/// It is kept out of line so that the common path (a single Parquet page 
chunk,
+/// always short enough for one accumulator) is a plain loop without any chunk
+/// bookkeeping in front of it.

Review Comment:
   This is measured to make a difference. Generally see 
https://llvm.org/docs/Passes.html#licm-loop-invariant-code-motion
   
   A code sample is here: https://godbolt.org/z/56oWjTGWW
   
   LLVM will hoist some checking to outside, and it will trying to broadcast 
the `level`, and trying to simd comparing, which is useful for large loop but 
in-efficient for short loop. 
   
   I think keeping or reming it is both ok for me, if we think just making 
short inputs a bit slower is ok, we can just keep a `while (num_levels > 0)` 
loop.
   
   



##########
cpp/src/parquet/level_comparison.h:
##########
@@ -35,4 +37,56 @@ struct MinMax {
 
 MinMax PARQUET_EXPORT FindMinMax(const int16_t* levels, int64_t num_levels);
 
+// Out-of-line fallback for inputs larger than uint16_t::max() so the common
+// single-chunk path stays a plain vectorizable loop.
+template <typename Predicate>
+ARROW_NOINLINE int64_t CountLevelsIfChunked(const int16_t* levels, int64_t 
num_levels,
+                                            Predicate&& pred) {
+  constexpr int64_t kMaxCount = std::numeric_limits<uint16_t>::max();
+  int64_t total = 0;
+  while (num_levels > 0) {
+    const int64_t chunk_length = std::min(num_levels, kMaxCount);
+    uint16_t chunk_count = 0;
+    for (int64_t i = 0; i < chunk_length; ++i) {
+      chunk_count += pred(levels[i]) ? 1 : 0;
+    }
+    total += chunk_count;
+    levels += chunk_length;
+    num_levels -= chunk_length;
+  }
+  return total;
+}
+
+/// \brief Count the levels that satisfy `pred`.
+///
+/// Accumulates matches in a `uint16_t` so the compiler can vectorize the loop
+/// using 16-bit lanes without widening each comparison result to 64 bits.
+/// Inputs longer than `uint16_t::max()` are counted in chunks to avoid 
overflow.
+template <typename Predicate>
+inline int64_t CountLevelsIf(const int16_t* levels, int64_t num_levels,
+                             Predicate&& pred) {
+  constexpr int64_t kMaxCount = std::numeric_limits<uint16_t>::max();
+  if (num_levels > kMaxCount) {
+    return CountLevelsIfChunked(levels, num_levels, 
std::forward<Predicate>(pred));
+  }
+  uint16_t count = 0;
+  for (int64_t i = 0; i < num_levels; ++i) {
+    count += pred(levels[i]) ? 1 : 0;
+  }
+  return count;
+}
+
+/// \brief Count the levels equal to `value`.
+inline int64_t CountLevels(const int16_t* levels, int64_t num_levels, int16_t 
value) {

Review Comment:
   Here I found `CountLevelsIf` will be compiled into `DType` multiple times, 
which making output binary 0.5% more larger whichout linker's help. So I 
extract a short `CountLevels` here to help compiler able to not copy and paste 
this function everywhere 😅



##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1749,11 +1744,10 @@ class TypedColumnWriterImpl : public ColumnWriterImpl,
         *out_spaced_values_to_write = batch_size;
         *null_count = 0;
       } else {
-        for (int x = 0; x < batch_size; x++) {
-          *out_values_to_write += def_levels[x] == level_info_.def_level ? 1 : 
0;
-          *out_spaced_values_to_write +=
-              def_levels[x] >= level_info_.repeated_ancestor_def_level ? 1 : 0;
-        }
+        *out_values_to_write +=
+            internal::CountLevels(def_levels, batch_size, 
level_info_.def_level);
+        *out_spaced_values_to_write += internal::CountLevelsGreaterEqual(

Review Comment:
   This function is used only once, I don't know whether I should extract it or 
not ( DType was dispatch multiple times, so I think it's ok to extract it)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to