luis4a0 commented on code in PR #12107:
URL: https://github.com/apache/gluten/pull/12107#discussion_r3260487761
##########
cpp/velox/shuffle/VeloxHashShuffleWriter.cc:
##########
@@ -1315,6 +1316,47 @@ uint64_t
VeloxHashShuffleWriter::valueBufferSizeForFixedWidthArray(uint32_t fixe
return valueBufferSize;
}
+void VeloxHashShuffleWriter::accumulateInputEncodingCounts(const
ColumnarBatch& cb) {
+ // Only velox-typed batches expose per-child encoding; foreign batches
+ // (e.g. arrow round-trips coming from non-velox sources) will be flattened
+ // by `VeloxColumnarBatch::from` later and we'd undercount, so just skip
+ // them here rather than reporting a misleading "all flat" mix.
+ if (cb.getType() != "velox") {
+ return;
+ }
+ const auto* veloxBatch = dynamic_cast<const VeloxColumnarBatch*>(&cb);
+ if (veloxBatch == nullptr) {
+ return;
+ }
+ const auto& rowVector = veloxBatch->getRowVector();
+ if (rowVector == nullptr) {
+ return;
+ }
+ for (const auto& child : rowVector->children()) {
+ if (child == nullptr) {
+ ++inputEncodingCounts_[kInputEncodingOther];
+ continue;
+ }
+ switch (child->encoding()) {
+ case facebook::velox::VectorEncoding::Simple::FLAT:
+ ++inputEncodingCounts_[kInputEncodingFlat];
+ break;
+ case facebook::velox::VectorEncoding::Simple::DICTIONARY:
+ ++inputEncodingCounts_[kInputEncodingDictionary];
+ break;
+ case facebook::velox::VectorEncoding::Simple::CONSTANT:
+ ++inputEncodingCounts_[kInputEncodingConstant];
+ break;
+ case facebook::velox::VectorEncoding::Simple::LAZY:
+ ++inputEncodingCounts_[kInputEncodingLazy];
+ break;
+ default:
+ ++inputEncodingCounts_[kInputEncodingOther];
Review Comment:
Good catch — fixed in a208a78. Added a new `kInputEncodingComplex` bucket so
ROW / MAP / FLAT_MAP / ARRAY no longer get conflated with the rare-encoding
catch-all. `kInputEncodingOther` now only covers BIASED / SEQUENCE / FUNCTION
(and any future additions to `VectorEncoding::Simple`). New `complex` gtest
case exercises ARRAY + MAP children landing in the new bucket; existing cases
also assert `kInputEncodingComplex == 0` so the boundary is checked.
##########
cpp/velox/shuffle/VeloxHashShuffleWriter.cc:
##########
@@ -1328,6 +1370,22 @@ void VeloxHashShuffleWriter::stat() const {
}
LOG(INFO) << oss.str();
}
+ {
+ std::ostringstream oss;
+ oss << "Velox shuffle writer stat:InputEncoding";
+ int64_t total = 0;
+ for (auto v : inputEncodingCounts_) {
+ total += v;
+ }
+ for (int b = 0; b < kInputEncodingNum; ++b) {
+ auto v = inputEncodingCounts_[b];
+ oss << " " << inputEncodingName(static_cast<InputEncodingBucket>(b)) <<
"=" << v;
+ if (total > 0) {
+ oss << "(" << (100.0 * static_cast<double>(v) /
static_cast<double>(total)) << "%)";
Review Comment:
Agreed — fixed in a208a78. Added an `inputEncodingSkippedBatches_` counter
that's incremented on every early-return path in
`accumulateInputEncodingCounts` and printed at the end of the InputEncoding log
line as `SkippedNonVeloxBatches=N`. The encoding-bucket total + the skip count
now equal the `count` field on the surrounding `cpuWallTimingList_` lines, so
the two log blocks have comparable denominators.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
