[
https://issues.apache.org/jira/browse/IMPALA-15181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18116143#comment-18116143
]
ASF subversion and git services commented on IMPALA-15181:
----------------------------------------------------------
Commit b7a3727c9f066e9f74daa2b2df5eacab977db350 in impala's branch
refs/heads/master from Balazs Hevele
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=b7a3727c9 ]
IMPALA-15181: Smallify strings in GroupingAggregator's DeepCopy
Small strings are now smallified during the deep copy of
GroupingAggregator, and var len memory is not allocated for them.
Testing: validated that small strings are smallified when executing
TPC-DS Q67 using debug code.
Change-Id: I22311d9cb678874d5c53f19811fa335b646c8d6f
Reviewed-on: http://gerrit.cloudera.org:8080/24848
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Grouping aggregations are not making use of small strings
> ---------------------------------------------------------
>
> Key: IMPALA-15181
> URL: https://issues.apache.org/jira/browse/IMPALA-15181
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Assignee: Balazs Hevele
> Priority: Major
>
> TPC-H Q67 has a streaming pre-agg with several grouping aggregations. Some of
> the columns used for the keys are small strings that are eligible for the
> small string optimization (e.g. item's i_category and i_class). I added some
> debugging code to determine how much memory the small strings optimization
> was saving for grouping aggregations.
> {noformat}
> int GroupingAggregator::GroupingExprsVarlenSize() {
> int varlen_size = 0;
> // TODO: The hash table could compute this as it hashes.
> for (int expr_idx : string_grouping_exprs_) {
> StringValue* sv =
> reinterpret_cast<StringValue*>(ht_ctx_->ExprValue(expr_idx));
> if (ht_ctx_->ExprValueNull(expr_idx)) continue;
> if (sv->IsSmall()) {
> num_small_strings_++;
> small_strings_size_ += sv->Len();
> } else if (sv->Len() <= 11) {
> num_potential_small_strings_++;
> potential_small_strings_size_ += sv->Len();
> }
> varlen_size += sv->Len();
> }
> varlen_data_size_ += varlen_size;
> return varlen_size;
> }
> ...
> Status GroupingAggregator::GetNext(RuntimeState* state, RowBatch* row_batch,
> bool* eos) {
> RETURN_IF_ERROR(QueryMaintenance(state));
> if (!partition_eos_) {
> RETURN_IF_ERROR(GetRowsFromPartition(state, row_batch));
> }
> *eos = partition_eos_;
> if (*eos) {
> LOG(INFO) << "GroupingAggregator reached eos. Varlen data size: "
> << varlen_data_size_ << "\n"
> << "\tNum small strings: " << num_small_strings_
> << " Small strings size: " << small_strings_size_ << "\n"
> << "\tPotential small strings: " << num_potential_small_strings_
> << " potential small strings size: " <<
> potential_small_strings_size_
> << "\tCopyGroupingValues small strings: " <<
> copygroupingvalues_small_strings_;
> }
> return Status::OK();
> }{noformat}
> For TPC-DS Q67 on scale factor 5, this produced:
> {noformat}
> I20260714 15:37:53.094179 710844 grouping-aggregator.cc:274]
> 5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen
> data size: 55366901
> Num small strings: 0 Small strings size: 0
> Potential small strings: 1642739 potential small strings size: 10722894
> CopyGroupingValues small strings: 0
> I20260714 15:37:53.445653 710844 grouping-aggregator.cc:274]
> 5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen
> data size: 14319844
> Num small strings: 0 Small strings size: 0
> Potential small strings: 563651 potential small strings size: 3678131
> CopyGroupingValues small strings: 0
> I20260714 15:37:53.565832 710844 grouping-aggregator.cc:274]
> 5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen
> data size: 5262020
> Num small strings: 0 Small strings size: 0
> Potential small strings: 207104 potential small strings size: 1351759
> CopyGroupingValues small strings: 0
> I20260714 15:37:53.596243 710844 grouping-aggregator.cc:274]
> 5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen
> data size: 1322171
> Num small strings: 0 Small strings size: 0
> Potential small strings: 52036 potential small strings size: 339642
> CopyGroupingValues small strings: 0
> I20260714 15:37:53.624203 710844 grouping-aggregator.cc:274]
> 5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen
> data size: 1322171
> Num small strings: 0 Small strings size: 0
> Potential small strings: 52036 potential small strings size: 339642
> CopyGroupingValues small strings: 0
> I20260714 15:37:53.624967 710844 grouping-aggregator.cc:274]
> 5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen
> data size: 21408
> Num small strings: 0 Small strings size: 0
> Potential small strings: 1260 potential small strings size: 9688
> CopyGroupingValues small strings: 0
> I20260714 15:37:53.625154 710844 grouping-aggregator.cc:274]
> 5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen
> data size: 1670
> Num small strings: 0 Small strings size: 0
> Potential small strings: 217 potential small strings size: 1454
> CopyGroupingValues small strings: 0
> I20260714 15:37:53.625212 710844 grouping-aggregator.cc:274]
> 5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen
> data size: 59
> Num small strings: 0 Small strings size: 0
> Potential small strings: 10 potential small strings size: 59
> CopyGroupingValues small strings: 0
> I20260714 15:37:53.625252 710844 grouping-aggregator.cc:274]
> 5c4ef707060ddccc:5a24966100000004] GroupingAggregator reached eos. Varlen
> data size: 0
> Num small strings: 0 Small strings size: 0
> Potential small strings: 0 potential small strings size: 0
> CopyGroupingValues small strings: 0{noformat}
> Based on this, there are strings with length <= 11 that should be eligible,
> but small strings aren't being used by the grouping aggregator. I ran TPC-H /
> TPC-DS queries and didn't see any with small strings. Fixing this could
> reduce memory use (10MB for aggregator 0?) and avoid a read for the variable
> length buffer for the small strings (which is on a hot path).
> It's hard to notice this type of issue. I'm not sure how we could test this
> more methodically, but it's hard to tell if there are similar issues with
> other locations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
