Joe McDonnell created IMPALA-15181:
--------------------------------------
Summary: Grouping aggregations are not making use of small strings
Key: IMPALA-15181
URL: https://issues.apache.org/jira/browse/IMPALA-15181
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 5.0.0
Reporter: Joe McDonnell
TPC-H Q67 has a streaming pre-agg with several grouping aggregations. Some of
the columns used for the keys are small strings that are eligible for the small
string optimization (e.g. item's i_category and i_class). I added some
debugging code to determine how much memory the small strings optimization was
saving for grouping aggregations.
{noformat}
int GroupingAggregator::GroupingExprsVarlenSize() {
int varlen_size = 0;
// TODO: The hash table could compute this as it hashes.
for (int expr_idx : string_grouping_exprs_) {
StringValue* sv =
reinterpret_cast<StringValue*>(ht_ctx_->ExprValue(expr_idx));
if (ht_ctx_->ExprValueNull(expr_idx)) continue;
if (sv->IsSmall()) {
num_small_strings_++;
small_strings_size_ += sv->Len();
} else if (sv->Len() <= 11) {
num_potential_small_strings_++;
potential_small_strings_size_ += sv->Len();
}
varlen_size += sv->Len();
}
varlen_data_size_ += varlen_size;
return varlen_size;
}
...
Status GroupingAggregator::GetNext(RuntimeState* state, RowBatch* row_batch,
bool* eos) {
RETURN_IF_ERROR(QueryMaintenance(state));
if (!partition_eos_) {
RETURN_IF_ERROR(GetRowsFromPartition(state, row_batch));
}
*eos = partition_eos_;
if (*eos) {
LOG(INFO) << "GroupingAggregator reached eos. Varlen data size: "
<< varlen_data_size_ << "\n"
<< "\tNum small strings: " << num_small_strings_
<< " Small strings size: " << small_strings_size_ << "\n"
<< "\tPotential small strings: " << num_potential_small_strings_
<< " potential small strings size: " <<
potential_small_strings_size_
<< "\tCopyGroupingValues small strings: " <<
copygroupingvalues_small_strings_;
}
return Status::OK();
}{noformat}
For TPC-DS Q67 on scale factor 5, this produced:
{noformat}
I20260714 15:37:53.094179 710844 grouping-aggregator.cc:274]
5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen data
size: 55366901
Num small strings: 0 Small strings size: 0
Potential small strings: 1642739 potential small strings size: 10722894
CopyGroupingValues small strings: 0
I20260714 15:37:53.445653 710844 grouping-aggregator.cc:274]
5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen data
size: 14319844
Num small strings: 0 Small strings size: 0
Potential small strings: 563651 potential small strings size: 3678131
CopyGroupingValues small strings: 0
I20260714 15:37:53.565832 710844 grouping-aggregator.cc:274]
5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen data
size: 5262020
Num small strings: 0 Small strings size: 0
Potential small strings: 207104 potential small strings size: 1351759
CopyGroupingValues small strings: 0
I20260714 15:37:53.596243 710844 grouping-aggregator.cc:274]
5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen data
size: 1322171
Num small strings: 0 Small strings size: 0
Potential small strings: 52036 potential small strings size: 339642
CopyGroupingValues small strings: 0
I20260714 15:37:53.624203 710844 grouping-aggregator.cc:274]
5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen data
size: 1322171
Num small strings: 0 Small strings size: 0
Potential small strings: 52036 potential small strings size: 339642
CopyGroupingValues small strings: 0
I20260714 15:37:53.624967 710844 grouping-aggregator.cc:274]
5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen data
size: 21408
Num small strings: 0 Small strings size: 0
Potential small strings: 1260 potential small strings size: 9688
CopyGroupingValues small strings: 0
I20260714 15:37:53.625154 710844 grouping-aggregator.cc:274]
5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen data
size: 1670
Num small strings: 0 Small strings size: 0
Potential small strings: 217 potential small strings size: 1454
CopyGroupingValues small strings: 0
I20260714 15:37:53.625212 710844 grouping-aggregator.cc:274]
5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen data
size: 59
Num small strings: 0 Small strings size: 0
Potential small strings: 10 potential small strings size: 59
CopyGroupingValues small strings: 0
I20260714 15:37:53.625252 710844 grouping-aggregator.cc:274]
5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen data
size: 0
Num small strings: 0 Small strings size: 0
Potential small strings: 0 potential small strings size: 0
CopyGroupingValues small strings: 0{noformat}
Based on this, there are strings with length <= 11 that should be eligible, but
small strings aren't being used by the grouping aggregator. I ran TPC-H /
TPC-DS queries and didn't see any with small strings. Fixing this could reduce
memory use (10MB for aggregator 0?) and avoid a read for the variable length
buffer for the small strings (which is on a hot path).
It's hard to notice this type of issue. I'm not sure how we could test this
more methodically, but it's hard to tell if there are similar issues with other
locations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
